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Over its 30 years of space flight history, plus the nearly 10 years of design, development 
test and evaluation, the Space Shuttle Orbiter is full of lessons learned in all of its numerous 
and complex subsystems. In the current paper, only selected lessons learned in the areas of 
the Orbiter propulsion and power subsystems will be described. The particular Orbiter 
subsystems include: Auxiliary Power Unit (APU), Hydraulics and Water Spray Boiler 
(WSB), Mechanical Flight Controls, Main Propulsion System (MPS), Fuel Cells and Power 
Reactant and Storage Devices (PRSD), Orbital Maneuvering System (OMS), Reaction 
Control System (RCS), Electrical Power Distribution (EPDC), electrical wiring and 
pyrotechnics. Given the complexity and extensive history of each of these subsystems, and 
the limited scope of this paper, it is impossible to include most of the lessons learned; instead 
the attempt will be to present a selected few or “key” lessons, in the judgment of the authors. 
Each subsystem is presented separate, beginning with an overview of the hardware and their 
function, a short description of a few historical problems and their lessons, followed by a 
more comprehensive table listing of the major subsystem problems and lessons. These tables 
serve as a quick reference for lessons learned in each subsystem. In addition, this paper will 
establish common lessons across subsystems as well as concentrate on those lessons which 
are deemed to have the highest applicability to future space flight programs. 


Acronym List 


ABCD 

= 

Adiabatic Bubble Compression Detonation 

ALT 

= 

Approach and Landing Test 

APU 

= 

Auxiliary Power Unit 

ATP 

= 

Acceptance Test Procedure 

BF 

= 

Body Flap 

BSTRA 

= 

Ball-Strut Tie Rod Assembly 

COPY 

= 

Composite Overwrapped Pressure Vessel 

CB 

= 

Circuit Breaker 

ECO 

= 

Engine Cut-off (System) 

EPDC 

= 

Electrical Power Distribution and Control 

ET 

= 

External Tank 

EVA 

= 

Extra- vehicular Activity 

FCS 

= 

Flight Control System 

FCMS 

= 

Fuel Cell single-cell Monitoring System 

FDA 

= 

Fault Detection and Annunciation 

FORP 

= 

Fuel/Oxidizer Reaction Products 

GG 

= 

Gas Generator 

GGVM 

= 

Gas Generator Valve Module 
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GHe 

GH2 

GN2 

G02 

GSE 

ISS 

JSC 

kW 

LCC 

LH2 

L02 

MECO 

MMH 

MPS 

MPTA 

NDE 

NESC 

NSI 

NTO 

OMS 

OMRSD 

PGME 

PRACA 

PRSD 

PTFE 

RCS 

RSB 

R&R 

SCAPE 

SSME 

SSP 

SRB 

SSPTS 

STS 

TTA 

TVC 

WSB 

WSTF 


Gaseous Helium 
Gaseous Hydrogen 
Gaseous Nitrogen 
Gaseous Oxygen 
Ground Support Equipment 
International Space Station 
Johnson Space Center 
kilo watt 

Launch Commit Criteria 

Liquid Hydrogen 

Liquid Oxygen 

Main Engine Cutoff 

Monomethyl Hydrazine 

Main Propulsion System 

Main Propulsion Test Article 

Non-destructive Examination 

NASA Engineering and Safety Center 

NASA Standard Initiator 

Nitrogen tetroxide 

Orbital Maneuvering System 

Orbiter Maintenance Requirement Specification Document 
Propylene Glycol Monomethyl Ether 
Problem Reporting and Corrective Action 
Power Reactant and Storage Devices 
Polytetrafluoroethylene (Teflon) 

Reaction Control System 

Rudder/Speedbrake 

Removal and Replacement 

Self-Contained Atmospheric Ensemble 

Space Shuttle Main Engine 

Space Shuttle Program 

Solid Rocket Booster 

Station to Shuttle Power Transfer System 

Space Transportation System 

Thermo Chemical Test Area at NASA JSC 

Thrust Vector Control 

Water Spray Boiler 

White Sands Test Facility 


I. Introduction 

A brief description of each of the Orbiter subsystem will be provided, followed by selected historical events that 
shed some important lessons. A lesson learned summary table is provided at the end of each subsystem section 
for quick reference. Given the complexity and extensive history of each of these Orbiter subsystems, and the limited 
scope of this paper, the attempt is not to include all of the lessons learned; instead it will be to present a selected few 
or highlight lessons, in the judgment of the authors. Historical hardware anomalies or ones that remained as 
unexplained anomalies (UA), i.e., no root cause was determined, and provided little or no lessons, are not included. 
Detailed description of all historical Shuttle subsystem anomalies can be found in the SSP PRACA database. In 
addition, no attempt is made to explain detailed design solutions to specific hardware problems encountered over the 
years; rather the intent is to concentrate on general practical lessons, lessons learned that are common across 
subsystems as well as to concentrate on those lessons which are deemed to have the highest applicability to future 
space flight programs. 


Auxiliary Power Units (APU) subsystem: 

The Space Shuttle Orbiter has 3 independent monopropellant grade hydrazine fueled APUs, which through 
catalytic decomposition, transmit the mechanical power to drive the 3 hydraulic pumps. These, in turn, provide 
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hydraulic power to the flight control system actuators, main engine hydraulic valves and auxiliary end effectors. The 
APU subsystem consists of the APU unit, the hydrazine fuel tanks and distribution lines, injector water cooling 
system, drain system, exhaust duct and APU controller (Fig. 1-2). As with any new complex program, the large 
majority of the lessons learned occurred early in its history; from the development and qualification testing through 
the Approach and Landing Test (ALT) and early orbital flights. 



Figure 1: APU Line-replaceable unit Figure 2: APU Subsystem in Orbiter aft 

compartment 

The original APU procurement specification on which the contract was bid called for a 250 hour (hr) operating 
unit, however, during the development phase it was quickly realized that the capability was not there and it could 
only operate reliably for approximately 20 hrs, with extension to 40 hrs through periodic maintenance. In the 1992 
timeframe, a goal was established of 75 hrs, which was achieved after the upgrade from the baseline to the Improved 
APU (IAPU) design. 

There were several design problems during the early development of the APUs. Some of the major issues 
included: poor fuel pump performance and lubrication, turbine wheel blade and shroud cracking, gas generator life 
and hot-start capability, gearbox accumulator performance, turbine housing life/cracks, turbine failure containment, 
controller manufacturing, and Gas Generator Valve Module (GGVM) control valve seat life/cracks. Lessons learned 
from these development challenges can be found in Ref. 1. One example of this includes a change in the water 
cleaning process for the original GGVM. As part of a process improvement, a change was made to utilize de-ionized 
water. It turns out that this water caused leaching of the cobalt binder inside of the tungsten carbide seats, which in- 
turn resulted in the valve seats cracking after installation in APUs. This valve seat cracking was the cause of a 
launch scrub on STS-3 1 . Lesson here is to be extra vigilant when making process changes to critical hardware since 
changes can sometimes bring unintended consequences; in the case of the old APU GGVM, it turns out to be the 
wrong choice. 

The APU fuel tank isolation valve underwent several design changes over the life of the Orbiter Program. The 
original valve included a bellows design. During APU system level ground testing in the early 1970’s, the valve 
suffered a catastrophic detonation failure. Failure analysis concluded that the surge pressures created by the APU 
GGVM were high and frequent (1.5 to 3 Hz) that the bellows failed due to fatigue. The lesson here is that a bellows 
design is not compatible with a fluid system that experiences frequent, high (~720 psia) surge pressures. After 
several re-design iterations over the years, a new valve from MOOG Corporation was implemented in 1992, which 
eliminated all the previous critical design shortcomings. 

An explosive failure during qualification testing at the NASA JSC Thermo Chemical Test Area (TTA) vacuum 
chamber provided several lessons. This failure occurred during a hot restart attempt as a result of adiabatic bubble 
compression detonation (ABCD) at high temperatures. This test highlighted the importance of testing the complete 
integrated system in a vacuum and thermally representative environment, similar to those encountered during flight. 
This testing also helped define the cool down rates of the APU and showed that the APU could not be immediately 
restarted after shutdown without performing an actively cooled hot restart; uncovering a certification requirement 
shortcoming. In another development test a detonation occurred in the fuel pump seal cavity drain area between the 
fuel pump and gearbox. The carbon shaft seal was found to be fabricated from the incorrect material which 
deteriorated allowing metal to metal contact. This permitted the metal seal carrier to contact the rotating metal shaft 
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seal. This metal to metal contact developed a heat source of sufficient magnitude to ignite the air and hydrazine 
vapors within the seal cavity. The APU fuel pump was later redesigned as part of the Improved APU (IAPU) so as 
to avoid the possibility of any metal-to-metal contact even if the carbon shaft seal were to fail. 

During OV-101 ALT second manned Captive Active flight, liquid hydrazine discharged overboard from the 
APU drain line and was re-ingested into the aft fuselage. Approximately one third of the aft structure was covered 
with hydrazine which resulted in significant damaged wiring and insulation (e.g., Kapton). The root cause was the 
failure of the seal bellows of the fuel pump. It was later replaced with a face seal using an elastomeric secondary 
seal. In addition, the seal cavity drain port was relocated away from vent doors and all aft access doors were sealed 
with silicon gaskets. A lesson for future vehicles is to avoid placing fuel drain ports near access doors or assure 
adequate seals against propellant re-ingestion into vehicle compartments. 

During the first mission, STS-1, both APU #2 gas generators failed due to argon gas leak at a weld in the GG 
heater common cavity case. A crack in a weld allowed the argon gas to escape. Loss of the heat transfer gas causes 
the calrod heater element to overheat and melt at the break point. During qualification testing, these heaters had 
passed an evacuated test. It was not a long-term steady-state test and primarily consisted of cycling the heaters 
multiple times. Subsequent to the flight, all heater cases were functionally tested under vacuum and the resistance 
measured to monitor for any signs of leakage. Long term redesign consisted of separating the heaters into two 
individual cavities and performing functional vacuum leak tests as part of the acceptance process. This event 
demonstrated the need to verify that redundant elements are truly independent and improved inspection techniques 
and checkouts to verify critical systems. 

Also during STS-1 and STS-2, there were indications of bubbles trapped in the fuel feedline which resulted in 
abnormal GG pressures (bubbles continued to be observed until introduction of the Improved APU on STS-45). Gas 
bubble trapped inside the fuel lines can be adiabatically compressed to raise the temperature to detonation level, i.e., 
adiabatic bubble compression detonation (ABCD). An explosion during qualification testing had been attributed to 
this cause. Excessive dwell time at temperatures above 200°F caused fuel decomposition and bubble formation. As a 
result of this concern, great care was established at KSC to avoid introducing gas during fuel loading operations, 
e.g., vacuum fill. In addition, due to the normal decomposition of hydrazine in the fuel system at ambient 
temperatures, attempts were made to reduce the elapsed time between fuel servicing operations and launch. Low 
bubble point hydrazine filters also helped avoid bubble formation and abnormal gas generator pressures. 

During STS-2 pre-launch period, a launch scrub occurred due to APU #1 and #3 indicating high lubrication oil 
outlet pressures. The lube oil filter and drain passage were determined to be plugged with a pentaerythritol wax 
formation from the mixing of hydrazine and oil within the APU gearbox. Procedures were developed to flush the 
gearbox and replace the oil and filters. A redesign of the APU seal cavity drain system to eliminate the common seal 
between the oil and fuel resolved this issue. This event provided the following lessons: a re-design to solve a design 
flaw is more cost effective than continuous maintenance as a solution to an on-going problem, and compatibility of 
fluids must be well understood even if they are sealed from each other. 

Another problem that occurred during the early flights happened during STS-9 landing phase where two APUs 
suffered an explosion due to an injector tube crack which took everyone by surprise (Fig. 3). This required extensive 
research into the cause of the origin which was found to be stress corrosion cracking of the injector stem; primarily 
due to the environment to which the APU is exposed from the hydrazine decomposition products. All the 
qualification testing preceding STS-9 had not taken into account the long “down time” or exposure time between 
operation cycles. In order to make the most efficient use of time during qualification testing, the APU was operated 
for a specified number of hours to qualify it for the same amount of operational time, not taking into account any 
down time between flights. This was found to be a significant factor for this failure. Hence, as a lesson from this 
failure, the IAPU qualification program was designed so that a segment of the testing included the long down time 
between missions at KSC (approximately 3-4 months); while simulating the highly humid and salt corrosive Florida 
environment. 
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Figure 3: Damaged APU from STS-9 detonation, showing 
approximate location of cracked injector stem and hydrazine leak 


The discovery of fatigue cracks (roots and tips) in the original APU turbine disk blades led to the re-design of 
this component. Eight cases of blade cracking occurred on development units, qualification units as well as 
production flown units. Figure 4 shows a close-up of a fractured blade found during a routine overhaul inspection of 
APU#3 in 1987, removed from OV-104. When blades fail, loose pieces can cause extensive blade damage, high 
vibration, leading to shaft failure and possible uncontained wheel rupture. In all cases wheel crack damage was 
limited to blades and shroud, with all incidents being contained/safe, although some resulted in significantly 
increased vibration levels and noise. The investigation concluded that the 
cracks were due to high cycle fatigue, with some evidence of 
environmental interaction and blade rub. A redesigned 75 -hour APU 
turbine made from Rene 41, a Nickel-based high temperature and strength 
alloy, was implemented as part of the IAPU in 1992. Some lessons from 
this investigation as they apply to turbine wheel design, manufacturing and 
maintenance were: to incorporate a full-width shroud that supports the 
complete blade profile, perform periodic NDE (e.g., mapping via dye 
penetrant inspections), reduce the number of grinding operations (without a 
stress relieve) after the final heat treatment to prevent imparting residual 

stresses, use a controlled heating method (no use of torch), and specifying Figure 4: Close-up of fractured 
the weld inspection criteria after the finish is completed. APU turbine blade 



Just prior to STS-62, a rain shower at KSC resulted in some flooding within the aft compartment. The visible 
water was removed and some effort was taken to dry the insulation prior to flight. However, some residual water 
was trapped within the APU insulation. In-flight vacuum induced flash freezing of this water in the insulation 
caused hydrazine in the fuel line to freeze, resulting in fuel blockage. This is a failure mode nobody had ever 
considered or evaluated before and was induced by factors completely outside of the APU subsystem or vehicle. A 
lesson from this event is that if there is any indication of a water/fluid intrusion, assume that it could be entrapped 
under insulation and check. Areas should be identified within the vehicle where water may be trapped. Prior to 
rollout, vehicle vent doors and access doors that are vulnerable to rain intrusion should be closed. Insulation should 
be designed to avoid trapping water; e.g., drain holes. 


During STS-79 ascent an APU had an unexpected underspeed shutdown post-MECO due to a speed sensor 
failure. Destructive analysis of Magnetic Pickup Unit (MPU) #2 found an internal broken wire. However, a single 
speed failure sensor should not have caused an APU shutdown. Review of the wiring diagrams showed that MPUs 
#1 and #2 were installed with reversed polarity relative to the wiring specifications. The problem had not shown up 
in testing at the vendor because the wiring was different at the test facility; in fact, the test system was re-configured 
to match the incorrect production unit wiring. The wiring problem occurred when a new improved APU controller 
design was put in place for all vehicles before STS-51 flew in 1993. As a result of the mis-wiring, less redundancy 
existed than what was intended. It is interesting to note that the original baseline APU controller was not sensitive to 
mis-wiring since each MPU was dedicated to a single function with no voting logic. Multiple lessons can be drawn 
from this, such as: sometimes hardware upgrades bring unexpected sensitivities or consequences, ground test 
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hardware should have similar configuration as flown hardware, and having consistent callouts across all levels of 
hardware drawings. 

After STS-73 landed and the APUs were shut down, a decrease in fuel pump inlet pressure was observed with a 
corresponding increase in seal cavity drain pressure. Examination of the fuel pump showed large amounts of apiezon 
grease at the internal diameter of the seal and a film of apiezon grease on the fuel side of the mating ring, outboard 
of the carbon seal track. As a result of this, the application of apiezon grease was revised; using care with how much 
is applied. Also, the fuel pump was to be baked after any grease application to allow excess to drain off. The 
obvious lesson here is to avoid excessive use of grease to component interface seals. 

A large number of anomalies in the APU subsystem over the years have occurred within the thermal control 
system. All of these are well documented in the Shuttle PRACA database. These failures are typically categorized as 
one of three types: failed-on (e.g., failed-closed thermostat, heater short), failed-off (e.g., failed-open thermostat, 
wire open circuit), other (problems other than failed-on or failed-off cases). Of these, the most critical, and the one 
that presents the highest risk to the crew is the failed-on case. During qualification testing it was found that the 
heater “on” failure case allowed very little time for crew corrective action before unacceptable temperatures of 
300°F in the fuel system were reached. 

In the flight history of the Orb iter APUs, there are only 5 heater failed-on cases. One example was during the on- 
orbit phase on STS-41, after the crew reconfigured the heaters from A to B, the fuel bypass line temperature 
increased from 110° F to 258° F after surpassing the fault detection and annunciation (FDA) limit of 180° F in 
approximately four minutes. The heater was then immediately returned to the ’A’ system and operated nominally for 
the remainder of the flight. The unusually high temperature rise rate seen was due to a short to ground in that portion 
of the heater segment which was pinched during the replacement of the thermostat following a heater anomaly on 
STS-3 1 . This heater string was not functionally checked after the thermostat replacement (this was missed during 
ground test because the failure was not in the heater string with the defective thermostat - it was the one next to it 
that was not subjected to ground test). This highlights the importance of performing a full functional check after any 
critical hardware is replaced. Most of the other thermal anomalies on the APU have been attributed to snap-action 
thermostat failures which, due to their location on high vibration zones, result in progressive wear of the internal bi- 
metallic disc. These are typically replaced after each degraded performance in flight (e.g., narrowing control band of 
less than 6°). A lesson from this is that, in the long-term, design solutions are preferred over continuous 
maintenance. 

The APU original design specification was for 10 years or 100 missions, whichever occurred first. Recognizing 
those requirements, when the 10 year limit was approaching, the APU community began developing a 
comprehensive maintenance plan for the APU hardware. This maintenance plan provided an economical long term 
periodic maintenance procedure for the APU to assure acceptable operation over the years. It accomplished many 
objectives including: ensuring the inherent safety and reliability levels of the APU hardware, restoring safety and 
reliability of the hardware when deterioration had occurred, obtaining and maintaining historical data for future use, 
obtain reliability data for design improvements, minimizing costs and unscheduled removals. The maintenance plan 
was developed using the following methodology: evaluation of data from flight and ground checkouts in relation to 
APU health, APU part failure history (in-flight, ground checkouts, vendor), and evaluation of part redundancy and 
criticality. In retrospect, for reusable systems, it is imperative to develop a maintenance program for the most critical 
components within all the subsystem. The maintenance plan, including periodic inspections, was found to be a true 
preventative effort and was significant factor in continuing the operation of the APU reliably over the years. In the 
case of the APU subsystem, it was deemed that the APU unit was the most critical and no other components needed 
periodic maintenance. 

Another important lesson within the APU subsystem is the benefit of implementing fleet leader testing for 
critical hardware. By definition, the tested fleet leader hardware is of flight pedigree and is kept ahead of the fleet 
hardware in terms of total operational/cycle time. Establishing a fleet leader test program provides advanced 
indication of potential hardware degradation /problems, i.e., root out problems early. For the APU, fleet leader 
ground testing was performed on the APU unit as well as the APU fuel tanks. Selective non-destructive and 
destructive evaluations were performed over the years which allowed detailed assessments of component operational 
age life effects. 
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The APU lessons described above and several additional ones are summarized in Table 1. 

Table 1: Selected lessons learned in the APU subsystem. 


Problem Description 

Lesson Learned 

Hot re-start explosion during 
qualification testing. 

• Design solutions to correct the problem cause are preferred over solutions 
that treat the symptom. 

• Assure safe emergency capability with design changes and operational 
constraints. 

• Avoid starts when hydrazine fuel feed system temperature is above 200° F. 

• On hydrazine valves, ensure that hydrazine is well isolated from solenoid 
cavity, use segmented solenoid coils to reduce the chance of overheating 
due to shorts, and allow continuous safe operation. 

• Importance of high-altitude thermal vacuum testing. 

Detonation failure of fuel 
isolation valve due to internal 
bellows fatigue from surge 
pressures. 

• A bellows design is not compatible with a fluid system that experiences 
frequent, high (-720 psia) surge pressures. 

Gas bubbles in hydrazine system 
and potential for ABCD, e.g., 
indications of gas bubbles on 
STS-1 and STS-2. 

• Assure safe operational capability with operational constraints. 

• Design hydrazine fuel systems to mitigate the possibility of large surge 
pressures. 

• Limit the maximum soakback temperature and dwell time above 200°F in 
the hydrazine fuel system to minimize bubble formation. 

• Avoid metal-to -metal contact as ignition source in hydrazine. 

• Select a low bubble point pressure fuel filter to obviate trapping gas 
bubbles. 

• Reduce time between hydrazine fuel servicing operations and launch 
(<month). 

• Install burst disks on fuel drain system to reduce the potential of fuel line 
pressure decay. 

Gas generator roughness due to 
voids in the catalyst bed 

• Utilize compression spring on gas generator to take-up voids as catalyst loss 
occurs. 

Turbine wheel cracks. 

• Need to fully understand the operational environment. 

• Incorporate a full-width shroud to support blade profile. 

• Perform periodic NDE (e.g., penetrant inspections). 

• Reduce the number of grinding operations. 

• Use a controlled heating method. 

• Specify weld inspection criteria after the finish is completed. 

Contamination/wax formation in 
gearbox from hydrazine and oil 
mixing causing STS-2 launch 
scrub. 

• A re-design to solve a design flaw is more cost effective than continuous 
maintenance as a solution to an on-going problem. 

• Compatibility of fluids must be well understood even if they are sealed from 
each other; utilize dual seals if necessary. 

• Change out lube oil between flights to avoid contaminant accumulation. 

• Conduct qualification or fleet lead program to uncover time-related 
problems. 

STS-9 APU 1 and 2 fires during 
entry and APU detonations post 
landing caused by stress 
corrosion cracking in GG 
injector stems. 

• Reduce installation and removal stresses on critical materials; monitor if 
necessary. 

• Reduce residual carbon and stresses from manufacturing process. 

• Minimize leakage between firings by improved seals. 

• For reusable systems, implement periodic inspections (e.g., borescope) and 
maintenance plan based on critical parameters and expand as 
data/experience is gathered. 

• Locate instrumentation in the vicinity of high criticality leak sources to 
better enable fault detection, isolation and recovery. 
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Problem Description 


Lesson Learned 



Exhaust duct leakage during 
qualification test. 

Exhaust Gas Temp (EGT) 
failures 

Exhaust duct “E” seal leakage 

Diaphragm fuel tank ground 
operation 


Water cleaning process change 
resulting in GGVM valve seats 

cracks. 

On STS-1 Gas Generator Bed 
heaters A and B failed due to 
argon gas leak at a weld in the 
GG heater common cavity case, 
APU high in-flight vibration 


Fuel pump detonation during 
development test due to metal- 

to-metal contact 

Fuel Line Heater failures (Failed 
ON: STS-51B, STS-31, STS-34, 
STS-41, STS-121) 


STS-62 rain water intrusion in 
the aft resulting frozen fuel line. 


• Minimize contamination and rust in any form which can act as hydrazine 
catalyst. 

• Limit the number of starts on hydrazine injectors. 

• Change to chromized-layer injector also brought up the unintended 
consequence of nitriding/flaking of chromium layer. 

• Conduct qualification or fleet lead program to uncover time-related 

problems. 

• Establishing a life limit on relatively inexpensive hardware that did not meet 

qualification goals is sometimes more cost effective than re-design. 

• Deletion of frequently failing hardware that is no longer needed may be 

more costly than re-design to prevent failure. 

• Final leak checks should be done under the most realistic operating 

conditions. 

• Avoid the following: high rate of pressurization/depressurization causing 

thermal excursions, reverse pressurization (fuel pressure > gas ullage) 
causing diaphragm stretch, high pressure differential (-100 psid) in the 
normal direction (ullage > fuel). 

• Carefully review all critical process changes; changes could bring more bad 
than good. 

• Verify that redundant elements are truly independent. 

• Improve inspection techniques. 

• Verification of validity of operational limitations (i.e., redlines) and their 
revision is preferred over re-design, if proven acceptable. 

• Thoroughly review installation of control hardware to mitigate vibration 

effects; re-locate or add vibration dampening. 

• For rotating shafts used in fuel systems, avoid designs that could result in 
metal-to-metal contact as ignition source for a fuel detonation. 

• Replacement is an interim solution - design change should be implemented. 
Design solution is preferred over continuous maintenance. 

• Design propellant heating systems to mitigate the risk of heater failed ON 
condition or shorts, e.g., self-regulating or self-limiting heaters, solid-state 
thermostats, segmented coil valves, DC-DC converters for power and 
isolation, overtemperature thermostats close to the beginning of the heaters, 
installing temperature sensors to monitor fuel line temps. 

• Bi-metallic disc thermostat are susceptible to wear damage in high vibration 
zones. 

• Perform hill inspection and functional checkouts/retest verification 
following any hardware replacement or intrusive work and prior to flight. 

• For fail-on heater in hydrazine systems, little time is available for crew 
corrective action before reaching unacceptable temperatures of 300°F. 

• Closely review Criticality 1 failure modes to eliminate their failure potential 
or reduce failure probability to an acceptable minimum. 

• Use caution that thermostat installation does not lie in vicinity of 

neighboring components that can influence it to prematurely turn the heaters 
off or on. 

• Avoid rain water intrusion into critical systems; inspect if water intrusion is 
suspected. Close vent and access doors prior to vehicle rollout. 

• Areas should be identified within the vehicle where water may be trapped. 
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Problem Description 

Lesson Learned 


• Design insulation system to avoid trapping water. 

Fuel pump inlet pressure drop on 
STS-73. 

• Avoid excessive application of grease in component interfaces and, if 
possible, perform component bakeout post application. 

STS-79 speed sensor mis-wire 
resulting in APU shutdown 
during ascent. 

• Hardware upgrades can bring unexpected sensitivities or unique set of 
problems. 

• Ground test hardware had same incorrect configuration as flight units. 

• Consistent callouts across all levels of hardware drawings. 

Multiple loose APU gearbox 
pressure transducer installations 
(one found after 1 1 missions), 
due to installation procedure 
flaw. 

• Torque requirements are generic and impact all systems. Whenever a 
problem is encountered, check impacts to other subsystems. 

• Actual torque can be masked by interference during torque application. 
Verify torque procedure is adequately implemented and verified at each 
location. Build custom tooling if necessary. 

Importance of Fleet Leader 
testing 

• Fleet Leader testing program provides advanced indication of potential 
hardware degradation problems, mitigates costly and unscheduled 
redesigns, and reduces overall risk. 

Hydrazine seal cavity drain leaks 
and relief valve failures on 
multiple occurrences (leak, high 
and low crack pressure, fail to 
crack or relieve). 

• On hypergolic systems, addition of burst disc upstream of relief valve helps 
in reducing the fuel vapor migration and contamination to relief valve and 
possibility of seal degradation and leaks. 

• Helium can permeate through Teflon on flexhoses and drop the line 
pressure. 

• Periodically cycle relief valves to break the rust and reduce stiction. 

• IPA flushing and cleaning of components to reduce contamination. Clean all 
interface fittings to the same level as the valves. 

• When a dynatube fitting leaks, the flex hose should be taken off and 
reinstalled per drawing to assure proper seating before other action is taken. 

• Avoid fuel drain ports near access doors and/or assure adequate seals 
against propellant re-ingestion into interior compartments. 

Hypergolic Quick Disconnects 
(QDs) leaks 

• QDs should be selected/designed for ease of maintenance; reduce SCAPE 
operations. 

• Design solutions for frequently failing high-cost hardware are preferred 
over its repair/replacement. 

• Heaters might be needed on hydrazine tank ullage lines and QDs. 

• Periodic replacement of GSE filters. 


Orbiter Hydraulic Subsystem: 

The Space Shuttle Orbiter hydraulic subsystem 
consists of three completely independent hydraulic 
power generation and distribution systems 
designated as systems 1, 2 and 3. This subsystem 
provides power to actuate aerodynamic flight 
control surfaces (elevons, body flap, 
rudder/speedbrake), main engine gimbal and valve 
controls, external tank umbilical retractors, main 
and nose landing gear uplock and deployment, 
main landing gear brakes, and nose wheel steering 
(Fig. 5). 

As with most Orbiter systems, the hydraulic 
subsystem was certified through both tests and 
analyses. Qualification test included multiple test 



Figure 5: Orbiter hydraulic subsystem 
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stands such as: Flight Control Hydraulic Laboratory (FCHL), Hydraulic System Thermal Test (HSTT) and Main 
Propulsion Test Article (MPT A). FCHL included the coupled hydraulic/flight control subsystems for closed loop 
simulation. It was used in a formal test mode to certify the overall hydraulic subsystem satisfied all its internal 
design and external interface requirements. In addition, it was used in a developmental mode to obtain engineering 
data to support the various analyses, to investigate special and extreme operating conditions (off limit testing) and to 
support flight failure analyses. Lesson learned is the importance of performing a comprehensive and integrated 
mission-representative qualification testing with production flight hardware to fully understand all interfaces and 
their combined interactions. 


During STS-1, there were observed pressure spikes in hydraulic system 1 and 2 main pump outputs. Both the 
allowable pressure ripple of +/- 10% (+/- 300 psi) and the allowable pressure spike were exceeded. Pump ripple can 
excite the natural frequencies of hydraulic systems producing spikes, causing high stresses and shortening the life of 
the hardware. However, analysis also showed that pressure ripple is significantly damped by the downstream filter. 
The lesson is to perform hydraulic dynamic testing and analysis with a flight representative system. 

On STS-8, hydraulic circulation pump #2 failed during hydraulic thermal conditioning. The crew attempted 
multiple starts with no success; later found to be caused by a piece of metal jamming and shearing the shaft. While 
attempting to start the pump, the crew left the power to the pump on for about 13 minutes. As a result, the 
motor/inverter electronics burned. The pump startup procedure was modified to turn power off after 60 seconds if 
the pump fails to start. Lesson here is to power off systems that fail to start after a specified time to avoid burnout 
and potential collateral damage. 

During STS-76 ascent, a hydraulic system #3 leak developed, with the leak rate increasing to 1% per minute. 
The hydraulic system #3 main pump had been replaced during the STS-76 turnaround flow due to wire damage. 
There was also leakage from the high pressure line which was replaced as well. No high pressure leak checks had 
been done on the high pressure flexhose out of the pump. The system was redesigned later to move the check valve 
upstream to the pump outlet and allow high pressure leak checks with ground power. Lesson learned is to always 
verify the system integrity after any intrusive work is performed on the hardware and if possible, design the system 
for ease of integrity verification. 


Hydraulic system contamination and silting has been the culprit of many anomalies throughout the Orbiter 
history. Hydraulic contamination is always an on-going challenge for the hydraulic community. Strict Orbiter 
Maintenance Requirement Specifications (OMRS) and fluid requirements are followed to minimize hydraulic 
system contamination. This is especially critical for close tolerance parts, e.g., servo-valves, low- flow and/or close 
tolerance control valves. As an example, during pre-launch confidence tests, main pump pressure fluctuations 
occurred prior to STS-8 1 and STS-97 due to contamination. These are typically resolved by circulating the hydraulic 
fluid through the GSE and Orbiter system filtering systems. Unloader valve leakages have occurred on several 
missions due to hydraulic contamination. Some have resulted in significant mission impact due to the need to 
continuously operate the circulation pumps on-orbit to maintain bootstrap accumulator pressure (e.g., STS-2, STS-7, 
STS-8, STS-41D, STS-41G, STS 5 ID, STS 511). In addition, ET hydraulic umbilical actuators failed to retract due 
to suspected transient contamination on STS-1 19, STS-134 and prior to STS-130. It was concluded that 
contamination and silting accumulated on the ET actuators due to their low cycle use which allow particles to 
accumulate over time. Some lessons learned from these and other 
contamination-related hydraulic problems are: develop and implement a 
strict contamination control program, perform regular cycling of 
hydraulic components, particularly those with close tolerance internal 
parts, add a pre-filter to those critical components, regularly sample fluid 
and replace GSE filters. 

Hydraulic bootstrap pressure decay has occurred multiple times due to 
life and system contamination issues with the piston accumulators. The 
purpose of the bootstrap accumulator is to pressurize the hydraulic 
reservoir to maintain sufficient inlet pressure to the main pump for 
startup. On STS-50, an accumulator leaked on orbit due to nibbling of T- 
seal caused by extrusion of backup ring, resulting from long term 
exposure to pressure (Fig. 6). It resulted in continuous circulation pump 
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S/N 6107 gas seal 
t-ring, oil side 


Figure 6: Hydraulic piston accumulator 
seal damage 
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operation on-orbit. The majority of piston accumulators inspected have shown minor backup ring extrusion. As a 
result of this, a 4-year flight limit was imposed on all piston accumulators. A welded bellows accumulator-type 
design replaced the piston type design in 1999. No gas leakage has ever occurred from Orbiter (or SRB) bellows 
accumulator units. The bellows accumulator requires no servicing for the life of the accumulator. The obvious 
lesson here is the preferred use of welded bellows hydraulic accumulators to prevent leakage and loss of function. 


Just prior to STS- 109 launch, a routine inspection at the hydraulic main pump vendor found the pump’s port cap 
bolts to be dry-film-lubricated (DFL) rather than passivated, as it is required. This resulted in overstress of the bolt 
inserts due to the lower friction, and reduced structural margin, resulting in damage to the housings. Several pumps 
inspected and x-rayed had sheared aluminum housing threads and significant insert movement (Fig. 7-8). This could 
result in hydraulic pump port cap separation and hydraulic fluid leak resulting in loss of a hydraulic system. STS- 
109 launch rationale was given with the DFL bolts based on testing that indicated reduced, but sufficient margin 
remained to prevent leakage. A redesign of the port cap with passivated studs and self locking nuts resolved the 
concern. One lesson from this problem is to establish and verify adequate mandatory inspection points (MIP) in 
order to catch assembly errors. Another lesson is to carefully assess the use of helical coil inserts and the installation 
torque requirements, particularly in soft materials. This investigation highlighted the importance to design critical 
joints to be low maintenance with the capability to detect joint failure, easy to inspect, avoid moisture 
intrusion/corrosion, with high pullout strength and consistent preload (nut factor). 



Figure 7: Hydraulic main pump port cap 



Figure 8: Hydraulic main pump port 
cap showing displaced helicoil insert 
and sheared aluminum housing threads 


Contrary to commercial aircrafts, the Space Shuttle Orbiter did not initially establish a periodic maintenance 
interval for the flight control actuators. As most of the other Shuttle systems, the flight control actuators were 
certified to 10 yrs and 100 missions. Refurbishment of the elevon and TVC actuators occurred on a case-by-case 
basis, from the Program onset through the year 2001. These were driven by unscheduled removals for specific 
component repairs. Depending the typically limited disassembly, the actuators were repaired, rebuilt and underwent 
a partial acceptance tested prior to being delivered back as flight spares. Given the very few of these actuators that 
were disassembled during this period (i.e., <5 units) as well as the limited scope, very limited insight was obtained 
on the degradation of the internal component, including soft goods. In the case of the elevons and TVC actuators, 
the refurbishment effort was established as a result of a RSB Power Drive Unit (PDU) power valve failure during a 
Frequency Response Test (FRT) ground checkout, just prior to STS-101 in 2000. An extensive investigation into the 

root cause found that a spool stop on the power valve 
had moved due to accumulation of hydraulic oil behind 
the stops and subsequent incremental movement 
(backfill) with each operating cycled, restricting power 
valve spool movement (Fig. 9). This finding led to a 
subsequent re-design and refurbishment of similar 
actuators. In the interim, a unique ground tests (de- 
silting test) was developed to closely track and trend the 
movement of the spool stops between missions and 
allow sufficient margins above the established limits. 
Subsequent review of earlier data indicates that the 
anomaly was present two years before (1998), but there 
were no requirements to review the secondary spool 
differential pressures, other than during FRTs. Lessons 



Figure 9: Hydraulic actuator power valve showing 
displacement of 2-piece spool stop restricting 
movement. 


11 

American Institute of Aeronautics and Astronautics 



learned from this hydraulic actuator anomaly are: avoid spool stop inserts which can displace/unseat and restrict 
movement (e.g., design valve with single spool stop), always review critical parameter test data, establish a 
maintenance plan to assess hardware condition and refurbish damaged or degraded components. If a screening 
process is established it should be predictive rather than reactive. 

Another example of a displaced/unseated insert on a hydraulic component occurred in early 2006. A failure 
occurred during the Acceptance Test Procedure (ATP) elevated temperature test where the internal retainer became 
loose and displaced; it should be fully seated in housing with positive interference or shrink fit (Fig. 10). This 
particular isolation valve had flown 9 missions on OV-103 prior to this finding. Dimensional inspection found the 
retainer seat and two other areas of housing bore to be oversized. Oversized housing condition resulted in negative 
interference with the retainer at elevated temperature. This defect appeared to be the result of manufacturing error in 
machining process. All similar valves installed on all Orbiters, were x-ray inspected (7 per vehicle), and no retainer 
displacement was detected in any other valve. However, due to concerns with subsequent failure of valves, a 
periodic in-situ vehicle x-ray inspection was instituted, which proved to be successful. Lessons from this event are: 
for interference fits assure positive interference is maintained at all temperatures, design hydraulic valves without 
retainers or with positive retention, successfully using x-ray inspection for in-situ valve assessments in order to 
avoid costly removals and schedule impacts. 


Hydraulic Isolation Valve Retainer Displaced... Not Fully Seated 



Figure 10: Hydraulic isolation valve showing 

displaced/unseated retainer (shown in closed position). 

A hydraulic modification was proposed as an upgrade in the early 2000’s to move the quick disconnects (QDs) 
to a panel located in the vehicle mold line at the 50-1 aft compartment access door. Each ground turnaround Orbiter 
flow requires approximately 20 hours of powered hydraulic operations. Installation and removal of the hydraulic 
QDs into the aft compartment accounts for and average of 3 Problem Reports (PRs) per flow. The modification was 
cancelled just prior to implementation due to impacts to the Return-to-Flight schedule. However, relocation of the 
hydraulic QDs to the mold line would have: eliminated the need to carry hydraulic GSE into the aft compartment, 
reduced foot traffic and potential collateral damage in the aft, and reduced installation/removal time. 

There are advantages for future spacecraft to avoid the complexities and large infrastructure of the Orbiter 
hydraulic system, which requires continuous monitoring and maintenance to maintain system integrity and avoid 
leaks. Alternatives should be considered in the trade space of any future vehicle flight control system design such as 
utilizing localized hydraulics within the actuators, e.g., electro-hydrostatic actuators (EHAs), or eliminating 
hydraulics altogether, as in electro-mechanical actuators (EM As). EM As would reduce weight, increase reliability 
and safety, and decrease support and checkout requirements by eliminating both hydraulics and APUs. 

The hydraulic lessons described above and several additional ones are summarized in Table 2. 


Table 2: Selected lessons learned in the Orbiter hydraulic subsystem. 


Problem Description 

Lesson Learned 

Qualification tests with a flight 
representative and integrated 
hydraulic test system e.g., Flight 
Control Hydraulic Laboratory 

• Essential to perform a comprehensive and integrated mission- 
representative qualification testing with production flight hardware to 
fully understand all interfaces and their combined interactions as well as 
off-nominal testing. 
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Problem Description 

(FCHL). 

During STS-1, observed pressure 
spikes in hydraulic system 1 and 2 
main pump output. 

STS-8 hydraulic circulation pump 
#2 failure. 

During STS-76 ascent, a hydraulic 
system #3 leak developed, with the 
leak rate increasing to 1%/min. 


TVC actuator drift due to lock valve 
leakage causing surge pressures 

(waterhammer). 

TVC actuator commanded near 
hardstop causing oscillations. 


Piston accumulator leaks 


Unloader valve leaks resulting in 
accumulator pressure decay. 

Hydraulic system contamination 
and silting causing multiple 
failures/leaks. 

Main Pump port cap to housing 
pulled out inserts due to incorrect 
use of bolts prior to STS-109. 


Hydraulic isolation valve retainers 
unseated/displaced, restricting 
movement. 


Permaswage reducer tee fitting 
cracks and leaks. 

RSB PDU spool stop displacement 
prior to STS-1 01. 



• Perform hydraulic dynamic testing and analysis with a flight 
representative system. 

• Verification of validity of operational limitations and their revision is 

preferred over re-design, if proven acceptable. 

• Power off systems that fail to start after a specified time to avoid 
burnout and potential collateral damage. 

• Shuttle bus impedance (60+ ft wire runs) mitigated impact. 

• Always verify the system integrity after any intrusive work is performed 
on the hardware. 

• Even though a leak check is performed with ground power, subjecting 
the component (e.g., flexhose) to vibrations could reveal a leak that 
could go undetected. 

• Design the system for ease of leak checking and integrity verification. 

• Match actuator command and position before hydraulic pump start 
(<2°). 

• Avoid actuators with mechanical feedback or avoid commanding 
actuators near mechanical stops. 

• Careful selection and matching of servovalves; reduce tolerance stack 

up- 

• Welded bellows accumulators have been better than piston accumulators 
for preventing leakage. 

• Piston accumulator tend to self-generate more contamination. 

• Importance of contamination control. 

• Add pre-filter to components with close-tolerance parts. 

• Important to implement strict contamination control requirements. 

• Perform periodic de-silting procedures on critical hydraulic flight 
components (e.g., elevons, TVC). 

• Institute Periodic/preventative maintenance of critical components. 

• Establish and verify adequate mandatory inspection points (MIP) to 
catch assembly errors. 

• Careful assessment of helical coil inserts, particularly in soft materials, 
and torque requirements. 

• Design critical joints to be: low maintenance with capability to detect 

joint failure, easy to inspect, prevent moisture intrusion/corrosion, high 
pullout strength, and preload consistency (nut factor). 

• Be extra vigilant of internal valve retainers which can dislodge/displace 
and restrict movement. 

• For interference fits, assure positive interference is maintained and 
properly seated at all temperatures. 

• Use of periodic in-situ x-ray inspections to assess valve conditions and 

avoid costly component R&R. 

• Avoid side loads on hard lines during installation. 

• Utilize a more ductile material than titanium. 

• Perform periodic inspections. 

• Avoid spool stop inserts, i.e., design valve with single spool stop. 

• Establish a maintenance plan to assess hardware condition and refurbish 
degraded components. 

• Always review critical parameter test data. Subsequent review found 

that the anomaly was present 2 years earlier but there were no 
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Problem Description 

Lesson Learned 


requirements to review the data. 

• Freeze-plug technique successfully developed and implemented to 
perform hydraulic hardware removals on the pad without need to drain 
or isolate the interface lines. 

Complexity, serviceability and 
continuous leakage concerns with 
hydraulic systems. 

• Have all the hydraulic connections to the system at a mold line interface 
panel (This will preclude collateral damage to system components when 
having to install GSE for checkout and will reduce time of mates for 
system checkout). 

• Design hydraulic quick disconnects such that pressure can be relieved 
prior to mating. 

• Institute periodic/preventative maintenance of components (includes 
cleaning of subassemblies and replacement of seals). 

• Consider electro-hydrostatic actuators (EHA) or Electro-mechanical 
actuator technologies. 


Water Spray Boiler (WSB): 

The Water Spray Boiler (WSB) is a thermal control system that provides passive and active cooling capabilities 
to cool down hydraulic oil as well as the Auxiliary Power Unit (APU) lube oil. There are three independent WSBs in 
the Orbiter. Cooling occurs in the heat exchanger (container) which is comprised of a tube bundle flowing the two 
fluids to be cooled. Cooling is achieved through spraying water (or water/PGME mixture) onto the tubes, with the 
water vapor expelled overboard. The water tank is pressurized from a gaseous nitrogen (GN2) tank which is 
regulated down through a GN2 regulator/relief valve (Fig. 11). 




Figure 12: WSB temperatures during ascent 


Throughout the shuttle history, the WSB has exhibited over 36 cases of freeze-ups flown with water pre-load 
(>33% of all the Orbiter WSBs) that prevented cooling, resulting in impacts to safety, mission success and mission 
operations. These freeze-ups typically occured during ascent, prior to APU shutdown, as well as post- APU 
shutdown, as a result of the ice formation that forms in the container under space vacuum conditions. The high 
altitude environment promotes water-to-ice formation via flash- freezing conditions above an altitude of 126,000 
feet, where the ambient pressure drops below the water triple-point of 4 torr (Fig. 12). STS-3 had likely the most 
significant event where a WSB freeze-up resulted in the early (pre-MECO) ascent shutdown of an APU. Numerous 
unsuccessful design and operational attempts were taken to address the freeze-up problem. Implementation of an 
azeotropic mixture of Propylene Glycol Monomethyl Ether (PGME) and water successfully solved the problem. The 
lesson learned from this are: to avoid open to space vacuum cooling system where the cooling fluid pressure drops 
below its triple point, causing freeze-ups and blocking flow passages and orifices. 

The WSB single-stage GN2 regulator/relief valve units have had numerous failures over the years. These failures 
manifest as: fail to crack, fail to reseat, internal leakage (creep) and external leakage. The majority of failures have 
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been due to contamination, soft-goods degradation, and/or the quality of workmanship. Metallic and/or non-metallic 
contaminations, historically, caused the regulator internal and external leakage. Lessons learned from these include: 
selecting soft-good materials with higher strength and elasticity, imposing a tighter and cleaner ATP requirement at 
the vendor, and reducing the GN2 supply filter size, e.g., 10 microns. 

The WSB low pressure sensors and vent nozzle heaters have experienced multiple failures. Use of stycast, a 
ceramic-based material used as sealant, tend to develop minute cracks over time which degrades its sealing 
capability. A change from stycast to a glass-based sealer (silicon oxide) resolved the problem. 

Several GN2 isolation valves have failed decay checks over the years due to intergranular corrosion, 
predominantly limited to the weld. Investigation found the cause of the intergranular corrosion was the weld filler 
material (Cress-430) which develops larger structural grains. The corrosion started from the inside and propagated 
through the weld. Moist GN2 gets trapped within the void volume and initiates the corrosion process. Utilizing a 
bud-weld that fully penetrated along the tube cross section eliminates the void volume and allows no moist GN2 to 
be trapped. An inspection plan was developed (e.g., GN2 decay and GN2 quality maintenance checks) to identify 
the defected GN2 valves and remove them prior to becoming in-flight failures (IF A). Lessons learned from this 
includes: utilize weld filler materials that are resistant to moisture, periodically sample the GN2 for minimum 
moisture allowances, and continuously trend hardware data to uncover early any degraded performance. 

Since the beginning, the WSB aluminum heat exchanger containers have always had corrosion problems (Fig. 
13). Nine WSB containers were removed during the Orbiter flight history 
due to severe corrosion and external leakage, primarily at the inlet/outlet 
stainless steel connections. The average useful life was 12.5 years, based 
on actual corrosion time of aluminum containers in the fleet. Post landing 
sampling/analysis of container water revealed presence of aluminum 
oxide (A1203) corrosion byproducts. Using an aluminum water tank in 
an open system makes the tank prone to aluminum-oxide corrosion and 
bi-metallic galvanic effects. A galvanic corrosion couple is formed from 
the dissimilar metals (aluminum and stainless steel) in the presence of 
conductive water. Adoption of higher water quality control in 1990 and 
PGME/Water in 2001 slowed the corrosion process but did not eliminate 
it completely. A preferred solution would be to change the container 
material to a non-galvanic material such as titanium. 

The WSB heat exchanger temperatures sensors have presented a unique set of problems over the years. The inlet 
and outlet temperature sensors were not placed in the most optimum location such that they are not truly 
representative of the inlet and outlet temperatures. Lessons from this are: to install independent (dual-bead) sensors 
at each heat exchanger inlet and outlets, make the controlling sensor the same or in the same location as the 
displayed/do wnlisted sensor), periodically check the temperature sensors calibration curve drifts. 

The Water Spray Boiler lessons described above and some additional ones are summarized in Table 3. 


Table 3: Selected lessons learned in the Water Spray Boiler. 


Problem Description 

Lesson Learned 

Water Spray Boiler freeze-ups on 
more than 30 flights and >33% of 
units. 

• Avoid open to space vacuum cooling system where the cooling fluid 
pressure drops below its triple point; causing freeze-ups and blocking 
critical flow passages and orifices. 

• On open cooling systems, reduce the vent opening to maintain higher 
gas/liquid surface pressure. 

• Utilize a low vapor pressure cooling fluid. 

WSB regulator leaks. 

• Select soft-good materials with higher strength and elasticity. 

• Impose a tighter and cleaner ATP requirement at the vendor. 

• Reduce gas supply filter size (e.g., 10 microns). 

• Avoid designs that require relief valve actuation as part of nominal 
system operation, particularly during the dynamic ascent phase. 
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Figure 13: WSB heat exchanger 
containers showing evidence of 
galvanic corrosion 


Problem Description 

Lesson Learned 


• Continuously trend hardware data to uncover early any degraded 
performance. 

Stycast cracks on component seals 
(e.g., WSB low pressure sensor, 
vent nozzle heaters). 

• Replace the stycast with a less brittle glass base sealer, e.g., silicon- 
oxide, to avoid paths/cracks for leaks and corrosion. 

Water valve leaks (e.g., E-brite 
corrosion). 

• Physical inspect all critical welds. 

• Perform double wrench torque test to verify bi-metallic joint integrity. 

GN2 isolation valve failed pressure 
decay due to intergranular 
corrosion. 

• Utilize moisture-resistant filler materials on all welds. 

• Reduce/control moisture levels in GN2 supply tanks. Periodically 
monitor/sample moisture levels in GN2 systems. 

• Continuously trend hardware data to uncover early any degraded 
performance. 

Water heat exchanger container 
corrosion and external leakage. 

• Design water heat exchangers with galvanically compatible materials to 
mitigate corrosion. 

• Be careful in addressing corrosion problems by use of coatings. 

• Higher water quality controls. 

WSB heat exchanger temperature 
sensor limitations. 

• Install independent dual-bead temperature sensors at each inlet/outlet of 
heat exchangers. 

• Use same location for the controller temperature sensors and downlisted 
sensors. 

• Perform periodic calibration curve drift checks and, if necessary, re- 
pack with a high quality copper content and grease. 

Controller Overcool 

• Utilize digital controllers; this can also provide commonality with other 
subsystems. 

• Use a narrow band controller logic. 


Mechanical Flight Controls (MFC): 

The Orbiter Mechanical Flight Controls (MFC) consists of the rotary flight control actuators, i.e., four 
Rudder/Speedbrake (RSB) actuators and four Body Flap (BF) actuators, their source of mechanical shaft power 
from the RSB Power Drive Unit (PDU) and BF PDU, respectively, and the interconnecting drive shafts (Fig. 14). 

Post STS-28, two BF actuators were 
removed and sent to the vendor for tear- 
down and evaluation; this was driven by 
ascent video showing unexpected 
oscillation of the Body Flap surface. 

Concern with actuator overload damage 
drove a series of non-destructive 
evaluation of the internal gears. 

Unexpected wear of the planetary gears 
resulted in restricting the operational life 
to 22 flights for these original BF 
actuators. It also led to a subsequent 
redesign, which included: increased gear 
width and elimination of the Manganese 
Phosphate coating from all friction 
surfaces. Following implementation of 
this redesign in the 1990’s, the limited 
life restriction was removed. 

Beginning in 1991, inspection of the exterior surfaces of the BF and RSB actuators showed signs of corrosion 
pitting, primarily caused by the humid and salt environmental exposure at NASA KSC. Mitigations included efforts 
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Figure 14: Mechanical Flight Controls 




















to arrest the corrosion via the implemented of a standard repair process consisting of rust removal, measurement and 
documentation of the damage (e.g., mold impressions) followed by the application of corrosion inhibiting grease 
throughout the housings. This external corrosion was more prevalent on the BF actuators due to their location within 
the body flap cove area and propensity to accumulate water. In addition, in the year 2000, a permanent rework to the 
BF actuators was incorporated by the vendor which consisted of stripping all corrosion and applying a new base 
phosphate coat and Super Koropon primer on all external surfaces. This design change, implemented on an attrition 
basis, proved to be significant in reducing the rate of corrosion on BF external housings. 


As part of a fleet leader inspection effort, the fleet lead RSB actuator was removed from OV-103 in 2003. To 
everyone’s surprise, considerable internal damage was found in the form of planetary gear and ring gear wear and 
fretting corrosion in addition to grease degradation (Fig. 15). The internal grease was yellow-colored and had 
decreased lubricity, with some physical oil separation and chemical breakdown. Some local gear teeth areas had 
reddish-brown grease, chemically-degraded from the fretting corrosion process. These findings had a considerable 
cost and schedule impact to the Program in that all fleet RSB actuators had to be inspected and most of them 
replaced, with all RSB actuators on OV-104 requiring the imposition of a limited life given that no available spares 
existed. These findings also led to Program- wide increased emphasis on corrosion and aging vehicle concerns. A 
lesson here is for future space applications to utilize greases with extreme pressure (EP) additives necessary for 
efficient boundary lubrication, better out-gassing qualities, corrosion-inhibitors and improved properties against 
physical separation and chemical breakdown. 



Figure 15: RSB actuator disassembly, showing the planetary gear set corrosion damage. 


A review into the cause of the RSB actuator fretting corrosion found that the RSB PDU, which is the input 
source to the RSB actuators, imparted a small oscillatory motion (limit cycling or dithering) during the majority of 
the ground operations periods. As viewed from the RSB PDU and drive shafts, it appears as a “washing machine” 
type of motion. Over the hundreds of hours of ground operation, this RSB PDU limit cycling caused an unintended 
grinding or fretting of the internal RSB actuator gear teeth. As a mitigation, RSB drive shafts were removed during 
most of the ground processing operations to decouple the source of the damage. In addition, the Program began 
carefully tracking the duration of all RSB ground operations. 


In addition, scheduled teardowns of RSB PDU at the vendor in 2003 and 2006 revealed many unexpected 
findings when the gearbox cavities were opened: significant gear scuffing, dark lubricating oil (MIL-PRF-83282), 
low oil level and significant particle contamination in the oil (Fig. 16). Silt and grease (Braycote 601) deposits were 

found in the brake cavities and throughout the units. These 
units had flown 25 missions. The internal gearbox on both 
the BF PDU and RSB PDU are sealed and isolated from the 
MiL-H4332£2 Orbiter hydraulic fluid circuit. In essence, this was 

as newconditi. equivalent to driving an automobile for more than 20 years 

without checking and replacing the transmission or engine 
oil condition. This finding led to an immediate fleet wide 
inspection of all RSB and BF PDU units, replacement of the 
gearbox oil, and periodic inspections thereafter. The team 
developed in-situ NDE inspection techniques to accomplish 

. ^ . . these objectives and, in some cases, avoid cost and schedule 

Figure 16: RSB gearbox oil comparison; new 

vs. old oil after 25 missions 17 
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impacts. The obvious lessons from these findings are 
particularly on isolated fluid cavities that normally are 
not checked/sampled and are not continually filtered. 

Inspection of four Body Flap PDUs in 2005 
revealed that that their sealed gearboxes were 
overfilled with oil (-1600 cm 3 ), which could lead to a 
structural failure and loss of the system. The 
inspections were initiated as part of a fleet leader 
effort. Investigation found the cause to be internal 
leakage through the o-rings within the interfacing 
transfer tube (Fig. 17). There is no insight into this 
internal leakage post-assembly. It turns out that unless 
the brake is installed within 0.002 inches of its designed clocked location, the transfer tube o-rings and bores are not 
aligned concentrically, which may cause the o-ring in the brake housing to leak. Fed by high-pressure hydraulic 
fluid, this leak would drip directly into the sump of the gearbox raising its oil level. This leak may develop 
immediately or may develop over time as the o-ring ages in this non-concentrically loaded position. An assembly 
alignment fixture was build and all subsequent units did not show signs of internal leakage. The lesson here is to 
assure alignment of interfacing component seals, and conduct post-assembly leakage tests of gearboxes. 

All of these inspections revealed a considerable amount of useful information, particularly as it relates to internal 
and external corrosion damage, as well as the condition of the internal soft goods and lubricants. The main lessons 
learned from these findings were: importance of establishing a fleet leader inspection plan to assess the condition of 
the hardware over time and uncover any unexpected issues, and the importance of performing periodic inspections 
of all critical hardware for external/internal corrosion damage. These fleet leader and periodic inspections were 
supplemented by specialized checkout tests, analytical model predictions, as well as continuous improvements to the 
standard OMRS process. 

The Mechanical Flight Control lessons described above and several additional ones are summarized in Table 4. 


Table 4: Selected lessons learned in Mechanical Flight Controls (MFC). 


Problem Description 

Lesson Learned 

External corrosion, internal wear, 
and corrosion fretting of RSB and 
BF actuators. 

• Perform periodic external and internal maintenance inspections on all 
critical hardware for corrosion damage. 

• Establish a fleet leader inspection plan to assess the condition of the 
hardware over time and uncover any unexpected issues. 

• Apply corrosion-preventive primer (e.g., Super Koropon) to all external 
hardware surfaces. 

RSB PDU gearbox discrepancies: 
gear scuffing, 

dark/contaminated/low oil level. 

• Perform periodic inspections on isolated fluid cavities that normally are 
not checked/sampled and are not continually filtered, e.g., gearboxes. 

RSB and BF actuators internal 
grease degradation. 

• Utilize grease with extreme pressure (EP) additives necessary for 
efficient boundary lubrication, better out-gassing qualities, more stable 
base oil, corrosion-inhibitors and improved properties against physical 
separation and chemical breakdown. 

BF PDU internal leakage 

• Assure correct alignment of interfacing seals and close-tolerance 
components to prevent leakage, e.g., transfer tubes. Build special 
alignment tooling if necessary. 

• Conduct post-assembly leakage tests, particularly of internal 
components with no available insight post- assembly. 

• Conduct periodic inspection of critical hardware cavities such as 
gearboxes. 


to perform periodic inspections on critical hardware, 


GEARBOX 


VALVE 



RAKE 


Figure 17: Body Flap PDU transfer tube 
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Main Propulsion System (MPS): 

The Space Shuttle Integrated Main Propulsion System (IMPS) consists of the ET, Orbiter MPS, and Space 
Shuttle Main Engines (SSMEs). Extensive GSE also exists to service, monitor, and maintain these elements. 
Overall, the IMPS is tasked with storage, conditioning, distribution, and combustion of cryogenic LH2 and L02 
propellants to provide first and second stage thrust for achieving orbital velocity. An overview of the Orbiter MPS 
portion of the IMPS is shown in Fig. 18. As indicated in this figure, nearly all of the MPS hardware is located in the 
Orbiter aft compartment. This includes the LH2 system, L02 system, GH2 system, G02 system, GHe system, GN2 
system, and other miscellaneous items. 



Figure 18: MPS Hardware Located in the Orbiter Aft Compartment 

A cracked BSTRA ball was found in Orbiter vehicle OV-103 during routine propellant feed line inspections of 
its 17-inch L02 manifold. Concerns associated with a cracked BSTRA ball included feed line structural failure, 
BSTRA joint malfunction, and SSME damage/failure from the ingestion of Foreign Object Debris (FOD) particles. 
Failure investigation work determined that the BSTRA ball cracks were caused by a defect related to an inadequate 
manufacturing process and poor quality screening. Silica inclusions from the sand cast process remained at the ball 
surface and created a thermal expansion mismatch with the Stoody 2 parent material, resulting in the formation of 
small thumbnail cracks during high temperature exposure associated with application of the Vitrolube coating. It 
was learned that the casting process can cause voids/inclusions in the parent material, leading to thermally-induced 
crack formation during subsequent steps in the manufacturing process or while in service. The Hot Isostatic Pressure 
(HIP) manufacturing technique can reduce or eliminate voids in the cast material as it cools. In addition, an 
inadequate acceptance testing and inspection process failed to detect the small cracks. This screening process should 
be made as robust as possible. 

The MPS flowliners smooth the flow over the gimbal joint bellows and extend bellows fatigue life by 
eliminating flow induced vibration (FIV) in the feedlines. In 2002, cracks were discovered in the 12 inch feedline 
flowliner slots (Fig. 19). Since these liners are located in the gimballing joint immediately upstream of the SSME 
low pressure pump inlets, particle liberation into the SSME would be catastrophic. The cracks caused by previously 
uncharacterized vibro-acoustic coupling of the flowliner and slots with high order surging and rotating cavitation 
generated by the SSME low pressure fuel turbopumps. These cracks were repaired by welding them closed while 
still on the vehicle, and the flowliner slots were then precision-polished to remove any residual micro-cracks from 
the original slot punching process. In future applications, it is important to fully characterize the environment, 
especially across element interfaces where relatively little may be known. Avoid the use of stamped flowliner slots, 
favoring machining if required at all, especially where debris liberation is critical. 
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Radial Crack 



Axial Crack 


Figure 19: MPS LH2 Feedline showing flowliner cracks 


During STS- 126 powered flight, the engine#2 Flow Control Valve (FCV) poppet broke across a portion of its 
flange (Fig. 20). This condition was detected via a change in inlet and outlet pressures though no command to 
change position had been sent. Since the poppet flange sees no impact loads, the cause was eventually deduced to be 
acoustic resonance caused by the high speed hydrogen gas exiting the valve. Special inspection methods were 
developed progressively as more was learned about the crack formation and propagation. Every flight valve tear 
downs and inspections were required to provide part of the continued flight rationale. It was learned that even 
structure with very high natural frequencies can be excited with acoustic inputs. Also learned was the fact that 
inspection methods need validation by complementary techniques, especially when cracks are very tight due to 
material properties and/or driving environment. 



Figure 20: STS-126 broken 
FCV poppet 

In the mid 1990’s, a fill and drain valve failed in mid stroke during vehicle testing. The cause was found to be 
design dimensional characteristics of the clutch mechanism which allowed loads to be applied between inner and 
outer clutch drivers, leading to eventual interference and galling. What was learned was that a postulated case, that 
of a structural failure in the valve train mechanism giving a false positive position indication while the valve failed 
to stroke, can actually occur. In this case, the valve failed to stroke fully when it galled, but the switch correctly 
indicated its position. However, the key on the indicator shaft sheared during subsequent testing, demonstrating that 
the position indicators can be driven fully to the desired state without a corresponding valve mechanism movement. 
This propensity exists because of a fundamental design deficiency in this type of valve where its position indicators 
are installed upstream of the propellant valve element. 


Composite Overwrapped Pressure Vessels (COPV) are currently used at NASA to contain high-pressure fluids 
in propulsion, science experiments and life support applications. In a COPV, the titanium or Inconel liner material is 
wrapped with Kevlar epoxy or Carbon/graphite overwrap. Prior to STS- 114, flight certification of the Orbiter 
COPV’s for stress rupture life was reviewed by the NASA Engineering and Safety Center (NESC) and identified as 
a flight risk issue. COPV fiber overwrap exhibits a failure mode known as stress rupture in which failure may occur 
at normal operating conditions and without warning, presenting a potentially catastrophic hazard. There has been 
extensive work in COPV’s over the years and numerous references are available on this subject (Ref. 7). The Orbiter 
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uses 24 Kevlar overwrapped pressure vessels for GHe and GN2 pressurant gas storage on MPS, OMS, RCS and 
ECLSS subsystems. As a result of the investigation, subsystem loading procedures were modified to limit 
temperatures and pressures during loading operations (e.g., staged pressurization), greatly increasing life. An 
important lesson was also to re-evaluate the original design assumptions with flight unit performance test data as it 
becomes available. 

“Mid-life” certification reviews were performed on the Orbiter in 2003-2004 to extend its design life to the year 
2020. During these reviews, many inadequacies in hardware certification were uncovered. Some of these were 
because of oversight at the time of certification; others were because the environments in which the hardware 
operates had changed with evolution of the Program. It is a good idea to periodically perform a review of 
environments for impact on hardware certification, using field and flight failure records as a guide on where to 
focus. 

Shortly after Return-to-Flight (RTF) following the Columbia accident, an Engine Cutoff (ECO) electrical 
problem threatened to ground the fleet because of fears of a common cause that could render the entire cutoff system 
inoperative. The “low hanging fruit” were investigated first, leaving the actual root cause of the problem to be 
investigated after much effort had been expended. The culprit was indeed found to be the feed through connector 
located on the ET, through which all four circuits pass. The defective connector was intermittently failing at 
cryogenic temperatures, a problem cured by soldering the pins to the sockets 3 . 

This portion of the paper considers only lessons learned during the design and certification phases of the Orbiter 
MPS hardware. Selected lessons learned in the MPS subsystem in the areas of component design and certification 
are summarized in Table 5. 


Table 5: Selected lessons learned in the Integrated Main Propulsion System. 


Problem Description 

Lesson Learned 

L02 feedline BSTRA ball cracks. 

• The casting process can cause voids/inclusions in the parent material, leading to 
thermally- induced crack formation during subsequent steps in the manufacturing 
process or while in service. The HIP manufacturing technique can reduce or 
eliminate voids in the cast material as it cools. 

• An inadequate acceptance testing and inspection process failed to detect the 
small cracks. This screening process should be made as robust as possible. 

LH2 feedline flowliner cracks. 

• Fully characterize the environment, especially across element interfaces where 
relatively little may be known. 

• Unintended consequences of fluid/acoustic environment, untested during 
certification. 

• Avoid use of stamped flowliner slots, favoring machining if required at all, 
especially where debris liberation is hazardous. 

• Proved success of “inspect and fly” methodology. 

Flow control valve poppet break. 

• Even structure with very high natural frequencies can be excited with acoustic 
inputs. 

• Inspection methods need validation by complementary NDE, especially when 
cracks are very tight due to material properties and/or driving environment. 

Position indicator mounting location. 

• Valve position indicators should be installed downstream of the propellant valve 
element rather than upstream at the actuator or in the drive train. 

COPY limited life due to stress rupture. 

• Use staged pressurization to reduce peak temperature seen during helium 
loading operations, greatly increasing life. 

• Important to re-evaluate the original design assumptions with flight unit 
performance test data as it becomes available. 
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Problem Description 

Lesson Learned 

Engine cutoff system investigation. 

• The approach of going after “low hanging fruit” does not always save money or 
time. 

• The most likely, most dangerous cause should be investigated first. 

• Common cause failures are very real and can seriously degrade system 
reliability. 

Oxygen material compatibility. 

• The use of Inconel and Monel was determined to be preferable to stainless steel 
based on the reduced propensity to ignite upon particle impact. 

• Testing should always be performed in a representative G02 environment to 
ensure that design sensitivities are identified early. 

• Careful consideration should be given to the use aluminum or aluminum alloys 
for piece-parts that experience relative motion in a G02 or L02 system. 

Fail-safe bellows. 

• If bellows must be used in a highly dynamic component or system, a fail-safe 
mode should be designed-in to avoid an undesirable condition in the event of a 
bellows failure. 

GHe check valve poppet jamming . 

• Careful attention should be paid to the Length-to-Diameter (L/D) ratio, radial 
clearances, and material selection when designing a sliding interface between 
valve piece-parts. 

• Perform representative testing, including high flow demand. 

Sharp edges and comers on component 
piece-parts. 

• Problem seen on several types of components. 

• Use generous radii at edges along sliding surfaces, at neck-downs, and on 
internal comer to prevent binding/jamming and reduce stress concentrations. 

Pre valve detent roller cracking. 

• A thorough tolerance stack up analysis should be conducted on all components 
containing moving parts to ensure that worst-case tolerances will not cause 
hardware malfunction or premature wear/failure. 

Prevalve main seal cracking due to 
slamming. 

• Design control preferred over operational control. 

Porosity in valve castings as a cause of 
external leakage. 

• When troubleshooting leakage, consider possibility that through-body leakage 
could be the source. 

• Design a seal gland to preclude erroneously satisfying leak check criteria. 

• Use Hot Isostatic Pressure manufacturing to eliminate porosity in castings. 

Minimize use of flexhoses and protect 
them from collateral damage. 

• For any spacecraft application, particularly on reusable vehicles, the use of 
flexhoses should be minimized. 

• Where flexhoses are deemed necessary, they should located away from areas of 
high personnel traffic and protected from potential collateral damage. 

Torque relaxation in cryogenic joints. 

• Design cryogenic joints with materials having similar coefficients of thermal 
expansion. 

• Consider geometry to minimize thermal gradients and relative thermal 
expansion. 

• Design the lines and flange joint insulation for maximum access to bolts. 

• Use Teflon-coated metallic seals rather than thicker non-metallic seals. 

• For non-metallic seals, perform a re-torque after 24 or 36 hours. 

Data time lag, gas mixing uncertainties, 
and inability to monitor all systems for 
hazardous gas leakage. 

• Use real-time hazardous gas detection system. 

• Ideally, use it on the ground as well as during powered flight in order to allow 
for crew response. 

Complications of inerting hydrogen 
systems in space. 

• The relief system should be a backup system by design, not one relied upon to 
prevent system or vehicle damage. 

• H2 tends to freeze on the cold walls of the system when exposed to vacuum, 
making expulsion of the commodity difficult near its triple point. Relatively 
warm GHe helps in sublimating the frozen H2, as does a lockup time before 
another period of opening the system to the vacuum of space. 

Subsequent failures caused by non- 
representative vibration testing. 

• Simulate flight installation secondary structure when performing random 
vibration qualification testing. 

Potential for inadequate initial 
certification and of program 
requirements “creep”. 

• Periodically perform a review of environments for impact on hardware 
certification. 

• Use field and flight failure records as a guide. 

Problems with hardware certified by 
similarity to other hardware in a “daisy 

• Avoid certifying hardware by similarity to other hardware which was itself 
certified by similarity unless a very good basis of testing is known. 
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Problem Description 

Lesson Learned 

chain”. 


Designed-in features or analytical 
margin are frequently not certified by 
test. 

• Perform burst testing, verification of redundancy, or operation with off-nominal 
inputs as required to verify design. 


Orbital Maneuvering System (OMS) and Reaction Control System (RCS): 

The Space Shuttle Orbital Maneuvering 
System (OMS) and Reaction Control System 
(RCS) are a pressure fed hypergolic propulsion 
system using monomethylhydrazine (MMH) 
as the fuel and nitrogen tetroxide (NTO or 
N 2 0 4 ) as the oxidizer. The Orbital 
Maneuvering System (OMS) provides orbit 
altitude adjustments for the space shuttle 
vehicle during the orbit phase, orbit insertion 
after ascent phase (OMS-2 burn), and the 
deorbit retrograde burn for the entry phase of 
flight. The OMS is located in two independent 
pods in the aft end of the Orbiter on either side 
of the vertical tail. Each OMS pod contains 
one OMS engine, a fuel tank, an oxidizer tank, 
and a helium tank, along with propellant feed 
lines, pressurization regulation, isolation valves, and other supporting equipment (Fig. 21). 

The Reaction Control System (RCS) is located in the two OMS pods and the forward module of the vehicle. The 
RCS provides the shuttle with orbit attitude control using multiple thrusters and entry control until aero surfaces can 
provide sufficient control authority for the shuttle. Each of the pods and module contains a collection of jets (38 
primary thrusters and 6 vernier thrusters in all), a fuel tank, an oxidizer tank, and two helium tanks, along with 
associated feedlines, pressurization regulation, isolation valves, and other supporting equipment. The Primary 
thrusters are rated at 870 lbf vacuum thrust and the Vernier thrusters are rated at 24 lbf vacuum thrust. 

Note at the end of this section, Table 6 documents the greater lessons learned in OMS/RCS. The Pilot Operated 
Valve (POV) for the primary thruster was the cause of many in-flight issues to which two topics (iron nitrate 
contamination and fuel valve extrusion) are highlighted below for the RCS system. 

Primary RCS Pilot Operated Valve (POV) fail-off or fail-leak due to iron nitrates 

The primary RCS thruster fail-off and fail-leak failure rate was increasingly high after 1988 return-to -flight and 
into the early 90’ s specifically with regard to the oxidizer valve. PRCS thruster redundancies were sufficient for the 
failures to date, however changing mission requirements and flight rules increased the criticality of the failure. This 
included contamination concerns associated with thruster valve propellant leakage during Space Station Mir 
missions and Low Z (aft firing and forward firing thrusters fired at same time provide a small translation in the Z 
axis) redundancy requirements for Mir rendezvous. High cost of repair and spares supportability were secondary 
issues. A Tiger Team gathered to address the issue. Gradual accumulation of metallic nitrates are believed to be the 
contributor to the majority of failures (Fig. 22). The failure mechanisms were grouped into two categories as such: 

1. Immediate Failure Trigger 

• Deserviced System - seal dry out resulting in leakage 

• Cold Temperatures - seal shrinkage resulting in leakage 

• Low System Pressure - decreased sealing load resulting in leakage 

• Off-Nominal Conditions during thruster R&R - eduction resulting in seal dry out and pulse purges (volume 
exchange operations) creating pressure surges and thruster leakage 

2. Long Term Nitrate Accumulation 


Primary 
Thrufttr (14) 



Figure 21: Forward RCS Module and OMS/RCS pods 
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• Moisture Intrusion - increased corrosion and nitrate formation 

• Sustained Low level leakage - nitrate formation at leak site 

• Oxidizer Quality - nitrate formation potential 



Figure 22: Iron nitrate on RCS 
POV pilot seat assembly 

KSC vehicle processing changes mitigated nitrate formation potential. Nitrogen tetroxide (oxidizer) is 
hygroscopic and will accelerate corrosion and formation of nitrates with moisture intrusion. Thruster R&R would 
require breaking into the system and attempting to keep moisture out. Rather than conducting a ‘hot’ R&R and 
risking moisture intrusion, procedure now directs a manifold drain and decontamination to the toxic vapor limit such 
that if air is introduced during thruster removal, nitric acid and iron nitrate formation is minimized. An internal 
active purge was designed to serve as a moisture barrier while all thrusters on a given manifold are removed for 
R&R. Seal saver installation in dynatubes is now required for all thruster R&R’s to minimize leakage. A new 
thruster purge system was developed to keep oxidizer vapors out of thruster chamber area as well as air moisture 
away from the valve seat since oxidizer vapors contribute to Fuel/Oxidizer Reaction Products (FORP) formation and 
later found to be the cause of fuel valve seat extrusion. Better thermal management for leakage and keeping lines 
hard filled during all phases of the vehicle flow processing were also implemented. Mole sieve was eventually 
implemented at KSC and WSTF to improve oxidizer quality. 

The combination of all the above procedure changes and added designs to mitigate moisture intrusion 
contributed to a significant decrease in thruster failures. However, as discussed in the next lesson learned subject, 
some changes increased the likelihood of another failure mode. The main lesson learned here is that a properly 
maintained hypergolic system can mitigate or nearly eliminate nitrate related ground and in-flight failures. 

Primary RCS POV fuel valve seal extrusion due to oxidizer vapor: 

Primary Thruster fuel valve extrusion was first discovered in 1995-96 during failure analysis following thruster 
failure during STS-68 and was expanded when other failures started occurring during thruster Depot processing. 
Conclusions from 1995-96 effort were not conclusive on the exact mechanism causing extrusion but did show 
correlation with a ground operations change. The extrusion mechanism was thought to be caused by either a one- 
time event high grade fuel/oxidizer reaction or a combination of low grade reaction with thermal cycling in a 
localized area at or near the pilot seat. In particular, WSTF thermal and exposure tests in -1995 had success in 
duplicating extrusion but results could not distinguish between these two causes. During the STS-68 failure 
investigation, it was also determined that due to the compressibility of GN2, fuel valves with extruded seals would 
exhibit slow main stage opening response or failure to open when actuated with GN2 while still passing response 
testing performed with liquid water or propellant; thus, fuel valve GN2 response testing was implemented as a 
Depot screen for fuel valve extrusion. 

Extrusion failures continued at a high rate after development of the GN2 response screen (~ 60 cases of fuel 
valve extrusion have been detected since initial failure, 8 per year average). Unfortunately, the screen was unable to 
prevent later in-flight failure and anomalies (10 flight failures through 2006). The occurrence of the last 6 in-flight 
failures during STS- 108 and -110 triggered formation of the Thruster Process Evaluation Team (TPET) to determine 
causes and improved corrective actions for fuel valve extrusion. The fact that fuel valve extrusion was not a problem 
for the first 15 years of Orbiter operations suggested that the cause was due to a change in thruster processing or 
usage. 
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In the early 1990’s, the Universal Throat Plug Assemblies (UTPA) were developed to provide a better seal 
against moisture intrusion in the chamber to protect against nitrate formation at the oxidizer valve. Nitrates were 
determined to be formed when oxidizer is exposed to moisture (as mentioned in the previous lesson learned topic). 
At the valve seat, nitrate accumulation typically resulted in either a sticky or a leaky valve. The UTPA worked well, 
but when coupled with increased throat plug installation time at KSC, a high oxidizer-vapor environment was 
inadvertently created in the thrust chamber. In late 1998, another processing change was made when a chamber 
trickle purge was implemented in an attempt to maintain a dry environment in the chamber and to reduce the 
oxidizer vapor concentration. This trickle purge was not always effective in practice. It was undersized, inefficient, 
and unreliable. The trickle purge was not active such as at the Vehicle Assembly Building (VAB) or when the pod 
or module was removed and located in the transfer aisle. Purge-down time allowed oxidizer concentrations in the 
combustion chamber to build rapidly. 

The TPET concluded that the fuel valve 
extrusion phenomenon was the result of a fuel- 
oxidizer reaction within the Teflon seal itself. 

Oxidizer vapor has been shown to readily permeate 
into PTFE (Teflon). Evidence of FORP within the 
PTFE pilot seals suggests that fuel may be forced 
into the semi-porous PTFE seal with regulated 
pressure and then reacts with the permeated oxidizer 
vapor. The reaction of the fuel and oxidizer would 
produce heat and evolve gas as a portion of the 
reaction by-products. It is therefore possible that 
evolved gas increases the internal void space of the 
PTFE by mechanical force, thus, causing expansion and a decrease in the density of the polymer. The extruding seal 
is not “walking out” of its retainer cavity, but instead the volume of the seal is increasing due to the larger internal 
voids. As a result, the seal extruded upward into the pilot stage flow path since that is the only unconstrained 
direction, as seen in Figure 23. 

The TPET investigation also resulted in the development of another extrusion screening technique utilizing the 
pilot stage electrical current trace recorded during the water response test. Review of the failed valves and new 
valves revealed that as the seal extruded upward, the shape of the current trace inflections change. In many cases, 
the valve may successfully pass the GN2 response test but exhibit an off-nominal current trace shape indicating 
some extrusion of the seal. 

Slow opening of the primary thruster fuel valves was caused by extrusion of the pilot stage Teflon seal. As the 
seal extrudes, the flow area of the pilot stage is reduced, effectively slowing the opening response of the valve. 
Extrusion is a slow process that occurs after weeks to years of exposure to oxidizer vapor during ground turnaround 
operations. Several corrective actions have been put in place, including reducing the allowable leakage of oxidizer 
valves and the new current trace extrusion screening technique. In addition, an improved chamber purge GSE design 
was implemented in 2006 allowing oxidizer vapor concentration to be maintained at a lower level. The simplicity of 
new purge GSE design reduced the cumulative purge downtime by reducing the time required to bring the purge up 
and down, increasing the ability of detecting a GSE leak, and allowing an active purge while in the VAB. 

The lesson learned here is that process is key when managing environments. The throat plug with purge 
assembly was designed primarily to help keep out moisture to address the nitrate formation and thruster failure 
issues, yet the plug also unintentionally trapped oxidizer vapor. An implemented trickle purge assembly meant to 
keep the chamber dry did not trickle at sufficient rates to rid the thruster chamber of vapor. This along with the 
inactive purge at certain points in the Orbiter processing flow allowed time for accumulated vapor to react at the fuel 
valve seat (this is another example of changes bringing unintended consequences). Tightening oxidizer leak 
requirements, trickle purge requirements and redesigning the purge assembly corrected the fuel valve extrusion 
while continuing to address the earlier nitrate formation issues. Developed current trace screen allowed for 
evaluation of seal extrusion without unnecessary valve disassembly at the depot. Since re turn-to -flight (STS- 114 to 
end of program) in-flight thruster failures due to nitrate formation or fuel valve extrusion were non-existent. 
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The OMS/RCS subsystem lessons described above and several additional ones are summarized in Table 6. 

Table 6: Selected lessons learned in the OMS/RCS subsystems. 


Problem Description 

Lesson Learned 

PRCS POV fail-off or fail leak due 
to iron nitrates. 

• A properly maintained hypergolic system can mitigate or nearly 
eliminate nitrate related ground and in-flight failures. 

• Need integrated hardware qualification testing with propellant, moisture 
and time exposure (Qualification was done incrementally and not 
integrated along with time exposure). 

• Implement preventative maintenance schedule to minimize in-flight 
failures by flushing all thruster valves periodically. 

• Full manifold R&R to prevent sympathetic failures. 

• Implement thruster chamber moisture control during ground operations 
since nitrogen tetroxide is hygroscopic and will generate nitrates. 

• Reduce low-level oxidizer leakage at the poppet/seat interface. 

• Improve the corrosion resistance of materials at the poppet/seat 
interface. 

• Minimize contact area between mating parts/surfaces to decrease 
potential nitrate -related stiction. 

PRCS POV fuel valve seal 
extrusion. 

• Thruster chamber moisture control hardware needs to provide sufficient 
ventilation of oxidizer vapor through use of a purge system. Purge 
system should be active through all phases of vehicle flow and flight 
preparation to prevent oxidizer vapor reactions with the fuel valve seat. 

• Thruster valve leak rate tolerances should be small to mitigate potential 
for thruster valve seat extrusion. 

• Use GN2 flow rate test to screen for failure. Consider developing 
current trace as part of screening method. 

PRCS combustion instability. 

• Although generally stable, instabilities were induced when helium 
bubbles injected into the fuel stream (propellant saturated with ullage). 

• Add stability screening test to thruster hot-fire ATP. 

• Add wire wrap as protection for thruster instability. 

FORP and Pc sense tube burn 
through resulted in near catastrophic 
failure on STS-5 1 . 

• Improve Pc tube design, e.g., avoid “S” shaped tube configuration or 
increase tube thickness. 

• Develop Pc tube flushing schedule to remove FORP. 

• Do not fire a suspect thruster during flight unless conditions are well 
understood and covered by qualification testing. Rather, inhibit and 
reprioritize thruster unless needed as a last resort. 

• Avoid very short firings; smaller impulse bit equates to less efficient 
combustion and more FORP formation. 

PRCS thruster injector relief radius 
and counterbore cracks due to 
cleaning and bake-out process. 

• Cleaning process and procedure development should consider material 
compatibility for all phases of the hardware production/preparation 
process (in this case, residual etchant remained in hard-to-rinse areas 
and the high temperature bake process caused material cracking). 

Vernier thruster reboost results in 
valve overtemperature. 

• Restrict vernier usage to specific reboost duty cycles. Develop go-no-go 
duty cycle criteria for long duration reboost requirements. 

Primary thruster injector 
intergranular cracking. 

• Elimination of pre-weld etch (Use of diamond coated cutting tool or 
titanium nitride coated cutting tool specified in machining operations). 

• Thorough review of thruster cleaning process to minimize cracks 

Flight failures occurring prior to 
ground test failures. 

• Periodically update fleet leader test article profiles to reflect flight 
usage. 

• Consider simulating similar ground environments and maintain fleet- 
like status with all hardware components and processes. This could have 
caught multiple failure types if process changes were captured in 
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Problem Description 

Lesson Learned 


fleetlead program. 

Primary RCS thruster L1L injector 
failure in ground test. 

• Avoid the unique set of ground test conditions that can lead to intra- 
manifold explosions such as: leaking thruster valves, leak duration, 
horizontal configuration, ambient pressure and thermal conditions. 

• Avoid firing leaking thrusters in ground testing. 

Sluggish regulator due to plugged 
sense tube 

• As soon as practical (i.e., post-landing), isolate the regulators from the 
propellant tank vapors or purge with inert gas to remove residual vapors. 

• OMS spiral restrictor tube design does not have contamination issues 
like the old RCS straight tube design. 

Quad check valve sticking issues 

• Unless the design calls for all joints to be hermetically sealed, assume 
oxidizer vapors will leak. 

Thrust chamber chips (R-512 
discilicide coating mechanically 
induced spalling). 

• Perform periodic inspections and track growth of known chips. 

• Injector film cooling provides longer primary thruster chamber life. 

• Characterize chips and develop acceptance rationale such that thruster 
nozzles do not need to be R&R for every chip. Columbium nozzles with 
disilicide coating is more robust than initially understood. 

OMS quantity gauging fuel probes 
problems. 

• Have altemate/redundant quantity gauging methods, e.g., ground 
loading gauging, in-flight engine firing time integration, PVT 
calculation. 

• Manufacturer’s expertise was with capacitor gauging sensors and not 
with complex electronic systems (totalizer). Recognize manufacturer’s 
expertise and limitations. Need right expertise for right technology. 

• More thorough acceptance test to screen out deficiencies in quality 
process. 

Propellant residue and 
contamination resulting in 
component leakage. 

• Install hardware to prevent propellant migration or utilize robust 
pressurization components that can endure propellant vapor migration 
and contamination abuse. 

• Improve program-wide contamination plan: buy precision cleaned 
components, clean and verify clean assemblies, verify that all fluids and 
GSE introduced to systems are clean. 

Dynatube fluid fitting repairs. 

• Seal saver design minimizes leakage at dynatube fittings interfaces. 

Zero-fault tolerant vernier thrusters 

• Vernier thrusters were an add-on and after thought for Shuttle. Vernier 
capability and fault tolerance should be evaluated for all potential 
missions on future vehicles. 

• Consider more efficient injectors as compared to the Shuttle single 
doublet vernier design to mitigate FORP. 

Ground operation issues: 

High ground processing timeline, 
cost, hazards (hot R&R, fire), 
supportability and obsolescence. 

• Closely track removal rates, repair times, spares counts, limited life 
hardware, and status of repair agencies. 

• Consider a non-toxic propellant architecture to decrease costs, hazards, 
and processing time. 

• Implement Integrated vehicle Health Monitoring system (IVHM) and 
expert systems. 

• Implement on-board OMS engine bi-propellant valve cavity drain/purge. 

• Relocate system components to the pod outer mold line and/or add 
quick component access doors to avoid the need to remove the whole 
pod for troubleshooting. 

• Change component and panel installation from welded to dynatube-like 
where possible. 

• Few area heaters (vs. large number of local heaters) are far easier for 
ground checkout. 
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Fuel Cells and Power Reactant Storage Distribution (PRSD) subsystem: 

The low temperature alkaline fuel cell (AFC) power plant consists of a 96-cell stack and an accessory section. 
The accessory section regulates reactant flow, manages the product water, regulates the temperature of the stack, and 
monitors multiple data parameters such as cell voltage, reactant flow rates, and potassium hydroxide (KOH) levels. 
There are three fuel cell powerplants on each Orbiter, located in the forward part of the midbody under the payload 
bay liners. All three fuel cells are capable of providing 2-10 kW nominal power (16 kW max) with voltage 
regulation between 27.5 - 32.5 VDC. These powerplants are each connected to one of the Orbiter electrical buses, 
Main Buses A, B, and C, and handle the load required by that bus as controlled by the Electrical Power Distribution 
Control (EPDC) subsytem. The reactants required by the fuel cell (oxygen and hydrogen) are stored in four or five 
(varying by mission requirements) Power Reactant Storage Distribution (PRSD) tanks, also located in the mid body 


Figure 24: Orbiter Alkaline Fuel Cell 



of the Orbiter. The water produced by the fuel cell is delivered to the Environmental Control and Life Support 
System (ECLSS) for use in both the cooling system and for crew consumption (Fig. 24-25). 



The first major in-flight fuel cell issue arose during STS-2 when FC1 had to be shut down less than five hours 
into the flight. At mission elapsed time (MET) +02:27, the crew reported that FC1 pH sensor reported off scale high; 
due to other pressing issues, the crew was unable to perform all of the contingency operations to confirm the pH 
level. Over the next two hours, FC1 ’s load-sharing decreased from a nominal 33% to 26%, indicating that the other 
fuel cells were picking up the load of the struggling fuel cell. At this point, flight controllers were concerned that 
possible KOH crossover might have occurred in the fuel cell and considered the fuel cell lost. The FC was then shut 
down and reactant valves were closed. Though the Orbiters are certified to land with only one working fuel cell, the 
failure of one results in a minimum duration flight 
(MDF). Once FC1 was shut down, the flight team 
became concerned that the isolated inter-cell 

potentials might be capable of water hydrolysis, 
producing free H2 and 02 and a potentially 
explosive situation; FC1 was restarted with the 

reactant valves still closed so that any remaining 

reactants in the stack would be consumed. This 
concern and safing procedure was new to the control 
team and had never been exercised during any 
mission simulations nor was it reflected in the 

onboard malfunction procedures for the crew. The 
team quickly learned that there should always be a 
plan to safely shut down and isolate any piece of 
hardware or even entire systems, particularly systems 
involving volatile reactants. 

Post-flight, it was determined that an H2 pump aspirator orifice (Fig. 26) was plugged with some contamination 
particle and that, without a properly working water separation system, the fuel cell had flooded. The aspirator was 
replaced with a recycle line, and filter screens were added to keep out contamination. Further, realizing that there 
was not enough insight into how the water separation system was functioning, the team added an H2 pump motor 
sensor that would indicate motor failure. Further, the team added a bus-tying procedure to the flight rules that would 
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Fig 26: H2 Pump Separator schematic; aspirator 
tubes highlighted in red square 


standardize tying two fuel cells and their buses together at any indication of a potential fuel cell failure. This not 
only prematurely configures the buses in case of a fuel cell shutdown, but also allows the flight controllers to assess 
the health of a fuel cell by monitoring the load sharing between multiple fuel cells. 

The second major issue occurred during the pre-launch activities of STS-4. FCl’s condenser exit temperature 
(TCE) shifted downward 8°F when the fuel cell loads were ramped up to their high loads at T-20min for launch. The 
mission continued with no adverse mission effects. Post-flight, it was determined that fluid entrapment blocked a 
relief slot in the TCE valve causing the low temperature reading shift. In order to diagnose these problems sooner in 
the launch-flow, a high load calibration was built into the fuel cell activation timeline; on all subsequent flights, all 
three fuel cells were taken to high loads (about 8kW each) for 30 minutes to ensure proper operation for launch. 

During the post-landing activities of STS-43, helium was ingested into the 02 line causing a performance loss in 
FCs 2 & 3. An incorrect GSE regulator setting allowed helium into the 02 system. The ground crew performed 
emergency power down procedures in order to shut down and safe the fuel cells before more damage could be done 
to the cells. These emergency procedures, however, assumed that the Orbiter was connected to ground power (as it 
usually is during pre-launch activities and again some hours after landing); in this case, the Orbiter had not yet been 
connected to a ground power supply. By shutting down and safing FC1, the only nominal fuel cell, the ground team 
left themselves without enough power on the Orbiter to throw the switches to isolate FCs 2 & 3 from their buses. 
Without sufficient power left to isolate, shut down, and safe the fuel cells, they continued to create power for the 
buses and consumed all of the reactants within the cells. The helium inside the cells continued to degrade the stacks; 
the fuel cells were lost and had to be removed from the Orbiter. 

This incident led to the “Best-Last” rule when powering down fuel cells: in order to ensure that the Orbiter 
retains enough potential (28V for the Orbiter) to isolate the fuel cells safely, the worst performing fuel cell should be 
shut down first, then the next worst, and finally the best fuel cell. Further, the ground team learned a lesson about 
evaluating emergency procedures to ensure that they always address the situation and configuration present. 
Emergency procedures should attempt to plan for all possible catastrophes and configure hardware to an absolute 
safe position. Proper procedure planning should aim to minimize the amount of unplanned reactions to unplanned 
events. 

In preparation for NASA’s plan to build an orbiting space station to which the Orbiter would dock, the fuel cell 
team brainstormed ways in which this configuration might affect the FC/PRSD subsystem; if the shuttle were to 
dock to some sort of larger station, it would lose its attitude flexibility and thus some of its passive heating 
strategies. The flight control team saw that this could create a potential problem if one of the fuel cells had been shut 
down and then needed to be restarted in this possibly “cold” configuration. From these misgivings, the STS-54 
detailed test objective (DTO 412) arose in which a planned FC shutdown/restart occurred. On flight day 6, FCs 1 
and 3 were purged in order to ensure their best performance while FC2 was shutdown. MN A and MN B were tied 
and connected to FC 1 while FC3 remained connected to MN C. After ten hours, FC 2 was restarted without any 
issues. It is stack-out temperature had decreased 60°, and the stack only required seven minutes of heating before it 
was ready for load. It was noted that during its shutdown, FC 2 sustained a 0.15V performance loss which, because a 
cold cathode activation cannot be performed on-orbit to recover the loss, was permanent for the rest of the mission. 
This important demonstration gave the fuel cell community confidence that a fuel cell could be shut down and 
restarted in a space environment and provided flexibility when planning future missions. 

From the first shuttle flight, a cell performance monitor (CPM) was used as a basic tool to gauge the health of 
the fuel cell stack. The CPM performed a self-test every seven minutes and reported the delta voltage for each of the 
fuel cell’s 32-cell substacks. If this delta value rose significantly during the flight, it would be an indication that 
there was uncontrolled mixing of hydrogen and oxygen within the cells, a condition called “crossover,” a mode of 
total failure for the fuel cell. For almost as long as the CPM has been a feature of the fuel cell system, it has passed 
along abnormal readings and false self-test failures. Various causes for failures including circuit shorts, cracked 
resistors, capacitor leaks, and unexplained anomalies were all noted. Despite the sporadic failures, the CPM often 
corrected itself, and good data regarding the fuel cell’s health was available at least intermittently. For instances of 
CPM failure, a procedure was developed to bus-tie the suspect fuel cell to a healthy one in order to monitor its 
performance via load-sharing information. 
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During the STS-83 mission, however, a small loss in performance in one of FC2’s substacks generated an 
abnormal CPM reading which could not be explained away as an occasional CPM failure. The delta voltage values 
continued to trend upward as the mission progressed and, without further insight into the cells’ health, crossover had 
to be assumed by the flight team. On the second day of the mission, FC2 was safed and then powered down while 
FCs 1 and 3 took over the loads for the remainder of the mission. With the loss of FC2, STS-83 became an MDF 
mission and landed just four days after launch. Post- flight, it was determined that there was no actual crossover of 
reactants in FC2 and that the CPM had only indicated the small performance loss experienced in a few cells. 
Because of this instance, the Fuel Cell single cell Monitoring System (FCMS) was developed and first used on STS- 
87. FCMS takes voltage measurements of each individual cell before fuel cell activation occurs all the way through 
ground-power disconnect just moments before launch. These values can give the pre-launch teams the confidence 
they need to launch in case of bad CPM readings. Once in orbit, the FCMS data has been taken at least once by the 
crew to ensure continued health of the cells and could be taken again at any time throughout the flight if needed. The 
lesson is that redundancy in performance metrics can be just as important as redundancy in system operations. 

In addition to changes that arose from issues encountered throughout the program, many technology and 
operations improvements were implemented as the demands on the fuel cell grew due to longer missions, more crew 
members, and dockings at the International Space Station (ISS). One of the first improvements in the program 
involved changing the stack from a 64-cell to a 96-cell stack by adding a third 32-cell substack. This addition 
increased the power capability from 10 kW to 12 kW, supporting a higher power profile for the Orbiter. Further, this 
addition added some redundancy in case of a loss of one of the powerplants as the other two fuel cells would be able 
to pick up more load between them. 

An operational improvement that took place over many missions was certifying the fuel cells to run through 
longer intervals without being purged. Initially, the flight rules specified that the fuel cells needed to be purged 
every eight hours in order to rid the stacks of the inerts that would build up and degrade performance. As flight 
controllers became more comfortable with the fuel cells’ performance through many days in-orbit, they began to 
stretch the limits of the purge interval over multiple flights. By the end of the 30-year program, the fuel cells could 
be purged every 96 hours or when a 0.2V decay had been observed, whichever came first. On flights when the 
Orbiter docked to station and received power from the ISS, the purge interval was raised to 120 hours or a 0.3V 
decay. By increasing the purge intervals, the team was able to decrease the complexity of operating the fuel cells as 
well as decrease some crew time from having to perform the manual purges. NASA learned that fuel cells can 
maintain steady performance for many days at a time and that performance drops of 0.2 or 0.3V can be almost fully 
recovered by a two minute pulse purge. 

The Space Shuttle was most often utilized in its last 1 0 years of operations as a construction vehicle that hauled 
pieces of the enormous ISS into orbit. Since much of its time on these missions was spent docked to the station, it 
seemed logical to design a way for the Orbiter to draw power from the massive solar panels and batteries of the ISS. 
These solar panels enjoy the renewable resource of sunlight as their power source whereas the shuttle’s fuel cells 
have a limited amount of cryogenic reactant on board. By being able to run the fuel cells at their minimum power of 
about 2 kW, the team would be able to increase their cryogenic margins and remain in orbit for 2-3 extra days. 
These added days allowed critical time for extra-vehicular activities (EVA) or other work on the ISS. 

The system that connected the Orbiter power system to that of the ISS was called the Station to Shuttle Power 
Transfer System (SSPTS). This system was able to increase the docked duration from about 6-8 days to 
approximately 9-12 days by transferring responsibility of up to 8 kW of the Orbiter’s load to the ISS. This power 
transfer was achieved by adding two power transfer units (PTUs) to the Orbiter which converted the 120VDC input 
power from the station to 28VDC expected by the main buses (Fig. 27). 

SSPTS capability was installed on OV-103 (Discovery) and OV-105 (Endeavour) and was first attempted with 
success on STS-118. Each flight of OV-103 and OV-105 afterwards involved using SSPTS to lengthen the docked 
missions. 

Since the ISS was able to handle most of the Orbiters’ power needs, the fuel cells ran at their minimum value 
(2kW) on-orbit. On some fuel cells that had higher operating hours, the sustaining heater was triggered due to a low 
stack output temperature. This heater nominally came on for 1-4 minutes until the output temperature raised enough 
to trigger an OFF signal. Some fuel cells saw zero sustaining heater cycles while on SSPTS while others saw up to 
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220 cycles. Age of the fuel cell and power requirement (lower power output meant lower operating temperature) 
were the two biggest factors relating to the number of sustaining heater cycles. 


The Fuel Cell and PRSD subsystem lessons described above and several additional ones are summarized in 
Table 7. 



Fig 27: SSPTS Electric Architecture 


Table 7: Selected lessons learned in the Fuel Cell/PRSD subsystem. 


Problem Description 

Lesson learned 

STS-2 FC1 shutdown 
following H2 
pump/water separator 
failure and FC flooding. 

• Bus-tying the suspicious fuel cell with a healthy fuel cell is the method for 
monitoring performance of the suspect fuel cell when one fuel cell picks up the 
load from another. 

• For vital systems, redundancy in monitoring the system’s health is as important 
as it is in the system itself. 

STS-4 condenser exit 
temperature downward 
shift. 

• Validate all instrumentation at maximum loads by running the fuel cells at high 
power for approximately 30min shortly after fuel cell activation. 

Pre STS-69 condenser 
exit temperature 
overheat. 

• Fuel Cell can not maintain normal conditions without cooling. 

• Procedures need to guarantee vehicle/FC coolant connections. 

STS-43 post-landing: 
fuel cells left on bus 
without water removal 
capabilities. 

• Operation timelines and procedures must be developed, executed and verified to 
avoid potential permanent damages to systems. 

• Cells can provide power when Fuel Cell is off. 

Fuel Cell exposure to 
high humidity. 

• Residual KOH on outer cell surfaces due to previous cell fill procedure can turn 
into a black sludge when the KOH absorbs water vapor. 

• Sludge causes shunt current effect across the cells but is reversible if not too wet. 

• Stack assembly procedures should take care to to contain KOH inside the stack. 

Fuel Cell shutdown and 
restart in mid-flight. 

• Redundant power systems need to be designed and proven in a space 
environment, i.e., shutdown/re start capability. 

STS-83 FC shutdown 
due to perceived 
crossover. 

• Residual oxygen in cells after an incomplete purge can create abnormal corrosion 
processes which requires single-cell voltage measurements to detect. 

• When vital instrumentation has a history of being erratic, a redundant 
measurement (such as FCMS data) must be available to rule out failures. 

• When failure can neither be proved, nor disproved in-flight, it is better to 
perform safing procedures and eliminate the risk of an unplanned failure. 
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Problem Description 


Lesson learned 


Port plugging (inability 
to clear inerts due to cell 
corrosion buildup). 

• All materials within the system need to be indentified for potential particles 
release and must include system Failure Mode and Effects Analysis (FMEA). 

• Carbonate formation inside 02/H2 inlet/outlet ports can arise from carbonation 
of fiberglass/epoxy FC frame. 

• Importance of maintaining system cleanliness. 

• Port plugging can be discovered during N2 diagnostic testing. 

• Can still fly with some port plugging so long as performance is monitored and 
purges are performed when necessary. 

Cell Performance 
Monitor (CPM) failures. 

• Certain EEE part failures are not detected byCPM self-check. 

• For suspected CPM failures, bus-tie to a good FC and monitor load sharing. 

Water relief line freeze 
possibility. 

• Operating environments need to be identified, implemented in design, performed 
and evaluated. 

• When new mission configurations arise (e.g., docking to a station), new 
problems may also arise which might require re-evaluation of risks. 

Cracking in fuel cell 
insulator plates (Noryl). 

• Any machining process should be evaluated for its effect on the material’s 
structural integrity. 

• Components should be routinely checked for degradation. 

STS-115 loss of phase in 
coolant pump and H2 
pump. 

• Three-phase electrical motors need to be evaluated and tested for being able to 
operate in two-phase. 

• Verified that the FC pumps could operate in flight on two phases; additional 
purges were necessary to maintain 2-phase operation. 

H2 vent capped during 
STS-38 checkouts. 

• System maintenance and pre-operation checklist procedures must be executed 
and verified prior to system startup. 

• FC regulators are unable to function if vents are capped. 

• H2 overpressure can damage the internal cells and irreparably damage the fuel 
cell. 

H2 pump current sensor 
sensitivity. 

• Unusual sensor activity can indicate a trend toward failure of the H2 pump 
motor, and thus a subsequent loss of the fuel cell. 

• When using an instrumentation measurement as a failure indication, make sure 
that all causes for output variations are understood and that procedures are in 
place for filtering out actual failure indications. 

Voltage pin sharing (cell 
voltage pin resistance 
fluctuations) 

• Add solder to the wire-to-pin crimp to complete conductive bond at connection. 

STS- 127 long sustaining 
heater cycle. 

• Effects of thermal environments must be included in design and demonstrated by 
testing. 

• Due to lower power requirements on SSPTS flights, the fuel cells sometimes 
require heating to return to operating temperature. 

• When the fuel cells are performing on the low end of their certification (2kW), 
sustaining heater cycles become a necessary component for operation: if they fail 
ON, the fuel cell needs to be safed immediately if the coolant system 
malfunctions in any way; if they fail OFF, the fuel cell’s load would need to be 
increased to maintain temperature, resulting in a cryogenic margin impact. 

Multiple 02/H2 
flowmeter erratic 
readings and failures 
(shifts in calibration, 
zero output, full-scale 
output, constant output). 

• Fluid flow path must be external to the electronic and instrumentation housing. 

• Sometimes, the cost and time required to redesign a troublesome component 
outweighs the benefit. 

• So long as a flowmeter still accurately indicates a purge, it will also accurately 
indicate a significant leak. 
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Electrical Power and Distribution and Control Wiring: 

The Electrical Power Distribution and Control (EPD&C) subsystem consists of a three-bus system that 
distributes electrical power (24V to 32V) from the fuels cells to the forward, mid, and aft sections of the Orbiter. 
The EPDC system consists of wires, circuit breakers, resistors, fuses, diodes, remote switching devices, and bus bars 
(Fig. 28). The EPDC system also includes three 120V, 400Hz, 3-phase, alternating current (AC) power busses, 
generated by sets of 750volt-ampere (VA) inverters from the main busses. 



The Electrical Wiring and Insulation Subsystem (EW&I) is made up of all electrical wiring and connectors 
external to subsystem boxes except for certain unique coaxial cables 
associated with communications equipment. This subsystem 
encompasses approximately 600 harnesses, 7,000 connectors, 120,000 
wire segments and 228 miles of wire (Fig. 29). 

On a series of missions (STS-75, STS-78, STS-79, STS-80 and STS- 
81) fuses associated with the cryogenic tank heaters, holding cryogenic 
hydrogen and oxygen, opened with no evidence of a short circuit or high 
currents. Investigation showed that the fuses were subjected to many 
cycles in which they carried currents close to their rating. This caused 
repeated thermal cycles which mechanically stressed the fusable link. 

This stress eventually caused the fuses to mechanically open (Kaymen- 
Halsey Effect). A design change was implemented that replaced the 5- 
amp fuses with 10-amp fuses and there were no reoccurrences. 

On STS-28, a short circuit occurred on a 20-gage cable powering the teleprinter through the aft crew cabin panel 
A15 utility outlet. The crew had reported seeing sparks and glowing embers coming from the cable at the panel. The 
utility outlet circuit breaker did not open to isolate the short because of the low amount of energy of the short. Data 
showed that the short continued to propagate for 1.5 seconds. Inspection of the damaged cable revealed that both the 
power and the return were melted into two pieces at the connector, and 3/4 of an inch of the Gortex cover was 
melted. The wiring was melted to 1 .2 inch from the wire tie on the stain relief up to the connector until the two wires 
were no longer touching; evident arcing had taken place. The connectors were exposed when the wires from a 
straight connector were bent at 90 degrees to fit into the recessed area, cracking the Kapton (polyimide) insulation. 

Industry experience with this phenomenon 
shows that the arc propagates when the heat 
generated by the arc chars (pyrolizes) the 
adjoining insulation which can now conduct 
electrical current and generate heat to cause 
further charring. In a wire bundle, the charring 
can spread between circuits, causing the 
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Figure 29: Examples of Orbiter wiring. 





failure of the whole bundle. Polyimide insulation was selected for shuttle wiring because of weight savings and the 
superior mechanical properties of polyimide. Covering polyimide insulation with a layer of PTFE (Teflon) reduces 
the likelihood of arc tracking (Fig. 30). In the shuttle, the risk of arc tracking was reduced by inspecting wire for 
damage that could start arcing and identifying critical circuits with high voltage differences and routing them in 
separate wire bundles so that arc faults would not affect redundant functions. 


On STS-31, during the power up for Flight Control System 
(FCS) checkout, the Air Data Transducer Assembly (ADTA) #3 
circuit breaker was closed but showed no signs of power. The 
circuit breaker was cycled five times. Three of these times, the 
LRU showed power, but following final closure, the power was 
off. The circuit breaker was cycled an addition five times, and 
ADTA#3 regained power and functioned nominally the 
remainder of the flight. There have been many (>10) 
occurrences of similar circuit breaker malfunctions (fail to 
conduct upon closure) during flight and on the ground. Multiple 
failure analyses have concluded that the failure mode is non- 
conductive debris interrupting continuity between the fixed and 
moveable contacts. This debris is generally small particles of 
the Circuit Breaker body’s molding resin, and is generated 
when the circuit breakers are cycled (Fig. 31). The circuit 
breaker construction does produce a small amount of contact 
wiping action which can clear contamination. A workaround to 
ground occurrences (implemented in the OMRSD) is to cycle 
the circuit breaker until it operates. Unfortunately this also 
generates more internal debris. The failure rate has been tracked 
levels. 



Figure 31: CBs used as power switches and 
redundant controls. Not a good practice. 


and is increasing but never reached unmanageable 


During STS-36 prelaunch ground operations, AC2 phase A inverter had numerous voltage spikes and current 
fluctuations in a 2-minute period. The inverter was removed and replaced and retest was nominal. The failed unit 
was sent to vendor for failure analysis. The problem was isolated to a lose connection caused by four loose screws 
(improperly torqued). Other suspected units were inspected and found to also have loose screws. These other units 
were removed and replaced. 


On the STS 51-B mission, when the right ET umbilical door was closed on orbit, a motor (B2) in the door drive 
actuator was inoperative. In postflight inspection, the problem was isolated to a loose/separated electrical connector. 
Most probable cause was failure to fully engage the pin/ramp locking feature when re-mating the connector. 
Difficult access and limited visibility due to connector location could have contributed to this condition. New 
standards of conformation of the connection were initiated. 


During the STS-65 mission (OV-102), the Spacelab experiment AC bus lost power when it was switched from 
the Orbiter inverter to the unpowered Spacelab experiment inverter. The 
Spacelab experiment AC bus was regained 3 minutes later by reconnecting it 
back to the Orbiter inverter. There was no impact to the mission; the AC bus was 
recovered. Personnel may have inadvertently bumped the AC bus switch on 
cockpit panel R7 to the experiment inverter position. 

On STS-93, a short circuit occurred in the AC system during ascent that 
caused the loss of the primary controller on the center main engine and the 
secondary controller on the left main engine. Post-mission inspection found a 
damaged wire in one of the payload bay wire trays. The failure analysis revealed 
that the conductor was exposed because of mechanical impact; something 
striking the wire harness against a screw that had raised burrs on the head (Fig. 

32). As a result of this failure, extensive vehicle wiring inspections were 
performed. As a result of the inspections, many other incidents were found of 
damaged wires and improperly protected wire harness. 



Figure 32: STS-93 wire short 
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The EPD&C, wiring and insulation lessons are summarized in Table 8. 

Table 8: Selected lessons learned in the EPD&C, wiring and insulation subsystems. 


Problem Description 

Lesson Learned 

Vehicle wire damage due to ground 
maintenance activity. 

• Important to perform specific detailed closeout inspection after work on 
cables or connectors. 

• Wiring is easily damaged by nearby work activity. 

• Training of personnel is critical both to prevent damage and to detect 
damage. 

• Wire chafe protection is important. 

STS-4 Heater fuse failure. 

• Repetitive thermal cycles on a fuse can cause it to mechanically open 

• Space and usage environments must be considered for fuse de-rating. 

STS-28 teleprinter short (arc fault). 

• Kapton wire insulation has unique failure mode, not anticipated to 
manifest in shuttle environment. 

• Arc faults do not necessarily cause currents high enough to trip circuit 
protection. 

• Fault propagated toward power source until distance between wire 
increased. 

STS-93 wire short. 

• Ground maintenance and vehicle modifications can cause damage like 
burrs on screw heads that lead to later wire damage. 

• Closeout inspections did not focus on inspecting the wiring. 

• Formalized and more clearly defined wire inspection criteria and 
processes. Perform periodic wiring inspections; particularly inspect for 
previously undetected wiring discrepancies. 

• Separate critical redundant wire runs and/or install barrier materials to 
protect against the effects of arc tracking. 

• Widespread use of chafe protection has greatly reduced wire damage. 

• Enhanced workforce awareness and training is paramount in ensuring 
the health of electrical wiring. 

Lightning monitoring system. 

• Low frequency bus voltage monitoring systems are inadequate for 
detecting lightning transients on vehicle power busses. 

• In order to detect lightning transients, monitoring system frequency 
response should be higher than lowest resonant frequency of vehicle 
power bus. 

• Low level signals easily couple onto bus from distant strikes, several 
miles away. 

• Pulse energy rather than voltage magnitude is a useful tool for assessing 
damage (Wunsch-Bell). 

Personnel switch bumps. 

• Redundancy of controls and inhibits important for critical functions. 

• Utilize switch guards. 

Circuit breaker problems. 

• Cycling circuit breakers recovers function. 

• Debris is created from internal wear on circuit breakers. 

• Avoid use of CBs as power switches and redundant controls. 

Bus voltage transients due to nearby 
lightning strikes 

• Solid state remote power controllers do not isolate loads from bus 
transients as effectively as electromechanical relays. This is true of 
MOSFET based controllers as well as transistor based ones. 
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Space Shuttle Pyrotechnics: 

Pyrotechnics are explosive devices that contain energetic mixtures of fuel and oxidizer compounds that when 
activated by electrical or mechanical initiation converts chemical energy to mechanical energy. Since, these units 
contain their own fuel and oxidizer they are ideal for use in the vacuum environment of space. Pyrotechnic devices 

were used throughout the Space Shuttle vehicles and on payloads 
because of numerous functions requiring high reliability, minimum 
weight, high energy density, precision timing and low cost. These 
devices provide the greatest amount of energy per unit mass, unit 
volume and dollars of any actuator that exists today. They can be used 
to perform in-flight applications and controlled tasks automatically or 
manually with the aid of simple electrical circuits or triggers. 

Some of the many functions that pyrotechnics performed on the 
Shuttle include booster ignition, stage separation, satellite 
deployment, fire suppression, thrust termination, landing gear 
deployment, crew escape, pressurization and control of fluid systems, 
equipment release and jettison, parachute deployment, opening and 
closing electrical circuits, battery activation, timing and delayed 
function, and cable cutting (Fig. 33). The fact that it’s extremely rare 
that a component from a flight qualified lot of Pyrotechnic devices 
has malfunctioned on missiles, rockets and spacecraft systems in the 
past is evidence that explosive and pyrotechnic devices can perform 
all the above mentioned functions, safely and reliably. To ensure 
safety, NASA requires that 10% of a production lot be subjected to 
environments and test fired at a range of temperatures. Also, units are 
serialized and have complete traceability from materials, 
manufacturing processes to final delivery. Although pyrotechnic 
devices are one-shot units, they are used redundantly (two units for a 
single function) meaning that the flight system has single fault tolerance. 


Figure 33: Shuttle pyrotechnic devices 


At least fifty (50) Government Furnished Equipment (GFE) pyrotechnics devices have been certified for human 
spaceflight by NASA-Johnson Space Center pyrotechnic engineers. These units include the NASA Standard 
Initiator (NSI), NASA Standard Detonator (NSD), several types of power pressure cartridges, mild detonating fuse 
and detonating cords, shear bolts, explosive bolts and frangible nuts, guillotines, reefing line cutters and other 
devices. About 140 NSIs were flown on each Shuttle flight. There were over 13,750 NSI fired during flight on the 
shuttle vehicles without a failure, with a reliability of 0.9999 at a confidence level of 95%. The qualification, 
certification, performance, and other pertinent information is contained in NSTS-08060, Space Shuttle System 
Pyrotechnic Specification. Selected lessons learned in the Shuttle pyrotechnic hardware are briefly described below 
and summarized in Table 9. 


During the STS-8 flight in 1983, the primary piston of the Nose Landing Gear (NLG) Extension Thruster was 
found on the runway threshold at Dryden Flight Research Facility (DFRC). It had been ejected from the housing of 
the thruster. Trajectory analyses determined that the piston had been fired from its centerline position for the nose 
gear release system without opposing force from the NLG strut and it had traveled between the two nose gear tires. 
Engineering studies dictate that the destruction of either of the NLG tires results in destruction of the 2nd tire and 
landing/rollout capability is lost. It was determined that o-ring leakage and “stickage” had resulted in the loss of 
“captured” air pressure behind the piston when the assembly is driven shut during NLG retraction at KSC and also 
that the single compressed internal spring was not physically adequate to forcibly extend the piston to a fully stroked 
safe position prior to firing. Without being able to hilly extend prior to firing when the NLG has deployed solely 
from gravity, the piston end-of-stroke-stabilizing-shoulder will shear off during firing and be ejected from the 
housing as was the case. This condition was not considered during the original design and was not required during 
qualification testing. The lesson learned is to make sure that the hardware design and qualification testing account 
for all potential flight conditions. In addition, never assume contained air pressure is available as an assist to 
pyrotechnic performance. 
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Multiple 3 /2 frangible bolt SRB hold down nut stud hang-ups have occurred during Shuttle flights. A firing skew 
between redundant cartridges caused small amounts of aluminum to be “shaved” from the SRB skirt bores when the 
redundant cartridge fired “late” and allowed the retained stud to drag through the bore prior to release from the nut. 
The NASA Safety Engineering Center (NESC) investigated the problem and designed a pyrotechnic Shielded Mild 
Detonating Cord (SMDC) crossover firing line between the redundant cartridges to reduce the firing skew and 
prevent stud hang-up. The lesson learned is to be sensitive to the firing skew present in all redundant pyrotechnic 
systems. 

Pyrovalve gas sampler failures have occurred during several Shuttle 
flights. These pyrovalves are part of the Orbiter Aft Fuselage Gas 
Sampler System (OAFGSS) which consists of two standalone racks 
located in the Orbiter aft compartment. Each rack contain three Vacuum 
Bottle Assemblies (VBA) and all electrical components required to fire 
the pyrovalves in flight. Post-flight analysis of the gas provides an 
estimate of Integrated Main Propulsion System (MPS) leakage by 
measuring aft fuselage hazardous gas concentrations during ascent. 

Typical failures result in the normally opened valve failing to close and 
trap hydrogen gas. Given the low criticality of the hardware, plus the 
fact are six gas samplers flown during each Shuttle flight, there was 
never any mission impacts due to valve failures to seat properly after 
firing/actuation. Failure analyses show that debris and manufacturing 
dimensional variations in the valves were the main contributors to the 
failures. (Fig. 34). 

During a lot acceptance tests of the NASA Common Pressure Cartridge, the pressure outputs were observed to 
be lower than required. An investigation revealed that changes in the manufacturing process of the propellant mix 
contributed to the lower pressures. The specific powder propellants (Hercules 6573 and 5808), had begun to 
deteriorate after several years of storage. Changes in the manufacturing processes had occurred that were not evident 
during the commercial certification of the material. 

During lot acceptance testing on the Orbiter/External Tank 2 !4 inch frangible nut, the nut failed to separate 
during a single booster cartridge firing. Investigations into the processing of the Inconel 718 alloy revealed that 
smaller grain structures had been obtained during raw material fabrication to meet the grain structures desired for jet 
turbine engine fan blades, the primary users of Inconel 718. Inconel alloy 718 is used to fabricate all NASA 
frangible nuts (%, 1 l A , 2 l A and the 3 V 2 inch). NASA revised their specification requirements to allow for an 
average grain size no finer than ASTM 7 using the radial forged (GFM) process. NASA also re-designed the booster 
cartridge to provide a greater margin of performance. The lesson learned is to be aware of raw material processing 
“refinements” over time which occur to satisfy other primary material users and to specify maximum as well as 
minimum material properties in applications that are sensitive like separation systems. Margin requirements are also 
sometimes underestimated. 


Table 9: Selected lessons learned in the Space Shuttle pyrotechnic subsystem. 


Problem Description 

Lessons learned 

STS-8 ejected Nose Fanding Gear Extension 
Thruster. 

• Verify that hardware design and qualification testing account 
for all potential flight conditions. 

• Never assume contained air pressure is available as an assist 
to pyrotechnic performance. 

3/2 inch frangible nut skew firing. 

• Be sensitive to the firing skew present in all redundant 
pyrotechnic systems. 

Pyro- valve poppet failure. 

• Debris in bore and/or manufacturing dimensional variations 
can cause pyrovalve to not properly seal after firing/actuation. 

• The NSI is an initiator and should not be used as a pressure 
cartridge, e.g., use NSI plus booster pressure cartridge. 

Propellant powder deterioration. 

• Changes in the manufacturing processes had occurred that 
were not evident during the commercial certification of the 
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Figure 34: Pyrovalve (post-fire) 
showing result of failure. 



Problem Description 

Lessons learned 


material. 

Inconel alloy 718 became stronger and harder 
to break. 

• Be aware of raw material processing “refinements” over time 
which occur to satisfy other primary material users. Need for 
control of material grain sizes (e.g., roll milling). 

• Need to specify both minimum and maximum material 
properties, particularly in applications that are sensitive like 
separation systems. 


Summary and Conclusions: 

The development, design, certification and operation of the SSP Orbiter power and propulsion subsystems have 
provided a wealth of lessons learned for future spacecraft applications, both re-usable and expendable vehicles. A 
subset of these have been discussed in this paper for the following Orbiter subsystems: APU, Hydraulics, Water 
Spray Boiler, Mechanical Flight Controls, Main Propulsion System (MPS), Fuel Cells and PRSD, OMS and RCS, 
EPDC, electrical wiring and pyrotechnics. There are both specific hardware lessons as well as general lessons that 
apply across all subsystems. Among the common relevant lessons are: 

1 . Importance of a thorough and integrated mission-representative qualification test program with production 
flight hardware to fully understand all interfaces and their combined interactions as well as the value of 
performing off-nominal testing. This will ensure that design sensitivities are identified early. Down time 
between missions or exposure time effects should be considered a well. However, it is important to keep in 
mind that no test can perfectly duplicate the actual operational conditions and good tests results are no 
reason for complacency. 

2. A re-design to solve a design flaw is more cost-effective than continuous maintenance as a solution to an 
on-going problem. However, verification of the validity of operational limitations is preferred over re- 
design, if proven acceptable. 

3. Importance of having a full understanding of the operational environment. 

4. System improvements, corrective actions for other problems, process changes, and/or mission requirement 
changes can bring unintended consequences or unexpected sensitivities. Recognize that requirements 
change with the changing space environments and those need to be accounted for in the future designs. For 
many Orbiter systems, the environment in which the hardware operates changed with the evolution of the 
Program. It is important to periodically review the operational environments for impacts to hardware 
certification, using flight failures as a guide on where to focus. 

5. Important to design critical systems to be fail-safe and to verify that redundant elements are truly 
independent. 

6. Implement a robust program- wide contamination control plan, including: buying precision clean 
components, clean and verifying clean assemblies, and verifying all fluids introduced to the system are 
clean before connecting. 

7. Importance of a robust and thorough Corrective Action process that avoids extensive latitude for accepting 
repeated failures without preventive corrective actions. 

8. Establish periodic and preventative maintenance, inspection and checkout plan for critical flight hardware 
to reduce the likelihood of failures. Checkout testing intervals should be set based on component criticality 
and system functionality requirements. 

9. Establishing a fleet leader test program. Fleet leader testing have offered the Space Shuttle Program great 
utility to work flight and/or hardware problem resolution, uncover problems early, GSE development and 
checkout and design changes independent of vehicle flow. This will be even more crucial for expendable 
programs where the flown hardware is not available for failure analysis. 

10. Trending of hardware operational flight and ground test data to uncover any out-of- family or degraded 
performance, well before they become critical, i.e., trend data for any shifts but not necessarily violating 
limits and comparing data from flight to flight. 

1 1 . Always verify the system integrity after any intrusive work is performed on the vehicle. 

12. Design systems for ease of leak checking and operational verification. 

13. Importance of maintaining an experienced/skilled ground crew. 
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In addition to all these, the importance of good record keeping cannot be stressed sufficiently enough. This is 
sometimes hard to judge; since there are many instances where information dating back well over 30 years ago has 
been extremely useful and applicable to understanding and solving current problems. All programs should establish 
an effort to retain, catalog and permanently store data such that can be easily retrieved in the future. 

Lastly, it is critical among all subsystems to maintain contacts with their hardware suppliers. Engineers should 
always be aware of the status of suppliers in case they are needed at any time. Periodic face-to-face contact, e.g., 
annual Technical Interchange Meetings (TIM) between the NASA customers and the hardware suppliers has proven 
extremely valuable over the history of the Space Shuttle Program. 

It is the hope of the authors that all the propulsion and power lessons presented in this paper be useful in future 
spacecraft vehicle applications. 
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Agenda: 


• Introduction and Approach 

• Subsystems Overview 

• Selected Subsystems Problems and Corresponding Lessons 1 ’ 2 

• Summary and Conclusion 

S Common lessons across subsystems 


'The accompanying paper provides a more comprehensive list of lessons and references. 
2 The symbol is used to indicate specific Lessons. 




Introduction 


There is a wealth of lessons in the ~40 year history of the Space Shuttle Orbiter. 

In the present briefing, only selected or “key” lessons are presented in the areas of 
Orbiter Propulsion and Power. These subsystems include: 

• Auxiliary Power Unit (APU) 

• Orbiter Hydraulics 

• Water Spray Boiler (WSB) 

• Mechanical Flight Controls (MFC) 

• Main Propulsion System (MPS) 

• Orbital Maneuvering System (OMS) and Reaction Control System (RCS) 

• Fuel Cell and Power Reactant and Storage Devices (PRSD) 

• Electrical Power Distribution and Control (EPDC), Wiring and Insulation 

• Pyrotechnics and Pyrotechnic Devices 

Approach: 

•Few or selected “key” lessons presented, in the judgment of the authors. 

- i.e., it is impossible to cover all or even most of all lessons. 

- The most popular historical failures do not necessarily provide the best 
lessons. Unexplained anomalies do not typically have lessons. 

•Focus on lessons that have potential application to future programs. 




Auxiliary Power Unit (APU) Lessons 


APU Subsystem Overview: 

• 3 independent monopropellant grade hydrazine fueled APUs. 

• Through catalytic decomposition, transmit the mechanical power 
to drive the 3 hydraulic pumps. 

• APU subsystem consists of the APU unit, the hydrazine fuel tanks 
and distribution lines, injector water cooling system, drain system, 
exhaust duct and APU controller 


Selected Problems and Lessons: 

• STS-9 APU 1 and 2 fires during entry and APU detonations post 
landing caused by stress corrosion cracking in GG injector stems. 
Reduce installation and removal stresses on critical 
materials; monitor if necessary. 

Reduce residual carbon and stresses from manufacturing 
process. 

Implement periodic inspections (e.g., borescope) and 
maintenance plan based on critical parameters and expand 
as data/experience is gathered. 

Conduct qualification or fleet lead program to uncover 
time-related problems. 

Change to a chromized-layer injector also brought up the 
unintended consequence of nitriding/flaking of chromium 
layer. 




ASSEfclSLY 


Hydraulic Pump 
Pad 


Lube System 


Fuel Filter exhaust 


GAS 
GENERATOR 


Orbiter APU Unit 



Damaged APU from STS-9 


See following pages for additional lessons 
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Auxiliary Power Unit (APU) Lessons 


Problem 

Lessons 

Hot re-start explosion 
during Qual testing. 
Adiabatic Bubble 
Compression 
Detonation (ABCD) 

Design solutions to correct the cause are preferred over solutions that treat the symptom. 
Assure safe emergency capability with design changes and operational constraints. 

Limit the maximum soakback temperature and dwell time above 200 °F in the hydrazine 
fuel system to minimize bubble formation. 

Importance of high-altitude thermal vacuum testing. 

Avoid metal-to-metal contact as ignition source in hydrazine. 

Reduce time between hydrazine fuel servicing operations and launch. 

Turbine wheel cracks. 

Incorporate a full-width shroud to support blade profile. 

Perform periodic NDE (e.g., penetrant inspections). 

Reduce the number of grinding operations, use controlled heating method, specify weld 
inspection criteria after the finish is completed. 

STS-1 Gas Generator 
heaters A and B failed 
due to gas leak in 
common cavity case. 

Verify that redundant elements are truly independent. 
& Improve inspection techniques. 

Water cleaning process 
change resulting in 
GGVM seats cracks 
and STS-31 scrub. 

Carefully review all critical process changes; changes could bring more bad than good. 

STS-79 speed sensor 
mis-wire and APU 
shutdown . 

Hardware upgrades can bring unexpected sensitivities or unique set of problems. 
Consistent callouts across all levels of hardware drawings. 
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Auxiliary Power Unit (APU) Lessons 


Problem 

Lessons 

Gas generator 
roughness due to voids 

Utilize compression spring on gas generator to take-up voids as catalyst loss occurs. 

Fuel Line Heater 
failures on many 
flights 

Design solution is preferred over continuous maintenance. 

Design propellant heating systems to mitigate the risk of heater failed ON condition or 
shorts (Examples: self-regulating or self-limiting heaters, solid-state thermostats, 
segmented coil valves, DC-DC converters for power and isolation, overtemperature 
thermostats close to the beginning of the heaters, installing temperature sensors to monitor 

STS-121 heater 
burn through 

fuel line temps). 

Bi-metallic disc thermostats are susceptible to wear damage in high vibe zones. 

Perform full inspection and functional checkouts/retest verification following any hardware 
replacement or intrusive work and prior to flight. 

Closely review Criticality 1 failure modes to eliminate their failure potential or reduce 
failure probability to an acceptable minimum. 

Exhaust duct leakage 
and sensor failures 

Establishing a life limit on relatively inexpensive hardware that did not meet qualification 
goals is sometimes more cost effective than re-design. 

Deletion of frequently failing hardware that is no longer needed may be more costly than 
re-design to prevent failure. 

Final leak checks should be done under the most realistic operating conditions. 

Hypergolic Quick 
Disconnects (QDs) 
leaks 

QDs should be selected and designed for ease of maintenance; reduce SCAPE operations. 
Design solutions for frequently failing high-cost hardware are preferred over its 
repair/replacement. 

Heaters might be needed on hydrazine tank ullage lines and QDs. 

Periodic replacement of GSE filters. 
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Orbiter Hydraulics Lessons 


Hydraulic Subsystem Overview: 

• 3 independent hydraulic power systems which provide power to actuate 
aerodynamic flight control surfaces (elevons, body flap, 
rudder/speedbrake), main engine gimbals and valve controls plus end 
effectors. 

Selected Problems and Lessons: 

• On STS-76 ascent, a hydraulic system #3 leak developed, with the 
leak rate increasing to 1%/min. 

Always verify the system integrity after any intrusive work is 
performed on the hardware. 

& Subjecting the component (e.g., flexhose) to vibrations could 
reveal a leak that could go undetected on the ground. 

Design the system for ease of leak checking and integrity 
verification. 

• Main Pump port cap to housing pulled out inserts due to incorrect use 
of bolts prior to STS- 109. 

Need adequate mandatory inspection points (MIP). 

Careful assessment of helical coil inserts, and torque 
requirements. 

Design critical joints to be: low maintenance, easy to inspect, 
prevent moisture intrusion/corrosion, high pullout strength, and 
preload consistency (nut factor). 



Orbiter Hydraulic Subsystem 



Pomp 

Housing 


Main Pump port cap pulled out inserts 
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Orbiter Hydraulics Lessons 


Problem Description 


Lesson 


Qualification tests of complex 
hydraulic system, e.g., Flight 
Control Hydraulic Laboratory 
(FCHL) 

On STS-1, observed pressure 
spikes in hydraulic system 1 and 2 
main pump output. 




* 


RSB PDU spool stop 
displacement prior to STS-1 0 1 



* 


Essential to perform a comprehensive and integrated mission-representative 
qualification testing with production flight hardware to fully understand all 
interfaces and their combined interactions as well as off-nominal testing. 


Perform hydraulic dynamic testing and analysis with a flight representative 
system. 

Verification of validity of operational limitations and their revision is preferred 
over re-design, if proven acceptable. 

Avoid spool stop inserts, i.e., design valve with single spool stop. 

Establish a maintenance plan to assess hardware condition and refurbish 
degraded components. 

Anomaly was present 2 years earlier (1998) during Orbiter Major 
Modification (OMM) at Palmdale, but there were no requirements to review 
secondary AP data other than during Frequency Response Test. Lesson is to 
always review of all critical parameter test data. 

Freeze-plug technique successfully developed and implemented to perform 
hydraulic hardware removals on the pad without need to drain or isolate the 
hydraulic interface lines. 


Hydraulic actuator power valve showing 
displacement of 2-piece spool 
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Orbiter Hydraulics Lessons 


Problem Description 

Lesson 

Hydraulic isolation valve 
retainers unseated/displaced, 
restricting movement. 

ft Be extra vigilant of internal valve retainers which can dislodge/displace and 
restrict movement. 

ft For interference fits, assure positive interference is maintained and properly 
seated at all temperatures. 

ft Use of periodic in-situ x-ray inspections to assess valve conditions and avoid 
costly component R&R. 

TVC actuator drift due to lock 
valve leakage causing surge 
pressures. 

ft Match actuator command and position (<2°) before hydraulic pump start. 

Piston accumulator leaks. 

ACCUMULATOR SHELL 

/ PISTON WEAR IS 

iW / ) SOURCE OF AL-2024 

K\[ ' / CONTAMINATION 

/ / ACCUMULATOR PISTON 

ft Welded bellows accumulators have been better than piston accumulators for 
preventing leakage. 

ft Piston accumulator tend to self-generate more contamination. 

Permaswage reducer tee fitting 
cracks and leaks. 

■ft Avoid side loads on hard lines during installation, 
ft Utilize a more ductile material than titanium, 
ft Perform periodic inspections. 

Hydraulic system contamination 
and silting causing multiple 
failures/leaks. 

Complexity/serviceability of 
hydraulic systems. 

ft Important to implement strict contamination control requirements, 
ft Add pre-filter to components with close-tolerance parts, 
ft Perform periodic de-silting procedures on critical hydraulic flight components 
(e.g., elevons, TVC, ET umbilical actuators), 
ft Institute preventative maintenance of critical components, 
ft Install hydraulic QDs interface at the vehicle mold line. 


Water Spray Boiler Lessons 


WSB Subsystem Overview: 

• 3 independent thermal control system that provide passive and 
active cooling capabilities to cool down hydraulic oil as well 
as the Auxiliary Power Unit (APU) lubrication oil. 


Selected Problems and Lessons: 

• WSB regulator leaks. 

Select soft-good materials with higher strength and 
elasticity. 

Impose a tighter and cleaner ATP requirement at the 
vendors. 

Reduce gas supply filter size (e.g., 10 microns). 

Avoid designs that require relief valve actuation as 
part of nominal system operation, particularly during 
the dynamic ascent phase. 

Continuously trend hardware data to uncover early any 
degraded performance. 

• Water Spray Boiler freeze-ups on more than 30 flights and 
>33% of units. 

Avoid open to space vacuum cooling system where the 
cooling fluid pressure drops below its triple point; 
causing freeze-ups and blocking critical flow passages 
and orifices. Utilize a low vapor pressure cooling 
fluid. 



Water Spray Boiler 



10 


Hydraulic 

fruc 






Water Spray Boiler Lessons 


Simulated Ascent Test of WSB: 
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Water Spray Boiler Lessons 


Problem Description 

Lesson 

Water valve leaks (e.g., E- 
brite corrosion). 

Periodically inspect all critical welds. 

Perform double wrench torque test to verify bi-metallic joint integrity. 

GN2 isolation valve failed 
pressure decay due to 
intergranular corrosion. 

Utilize moisture-resistant filler materials on all welds. 

Reduce/control moisture levels in GN2 supply tanks. Periodically monitor/sample 
moisture levels in GN2 systems. 

Continuously trend hardware data to uncover early any degraded performance. 

WSB heat exchanger 
temperature sensor 
limitations. 

Install independent dual-bead temperature sensors at each inlet/outlet of heat 
exchangers. 

Use same location for the controller temperature sensors and downlisted sensors. 
Perform periodic calibration curve drift checks and, if necessary, re-pack with a 
high quality copper content and grease. 

Stycast cracks on component 
seals. 

Replace the stycast with a less brittle glass base sealer, e.g., silicon-oxide, to 
avoid paths/cracks for leaks and corrosion. 

Water heat exchanger 
container corrosion and 
external leakage. 

Design water heat exchangers with galvanically compatible materials to mitigate 
corrosion. 

Be careful in addressing corrosion problems by use of coatings. 

Higher water quality controls. 

* S^ray Bar 


WSB Container Corrosion 
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Mechanical Flight Controls (MFC) Lessons 


MFC Subsystem Overview: 

• Consists of the Rudder/Speedbrake and Body Flap 
actuation system: 

• 4 Rudder/Speedbrake (RSB) actuators and drive 
shafts 

• 4 Body Flap (BF) actuators and drive shafts 

• 1 RSB Power Drive Unit (PDU) 

• 1 Body Flap PDU, respectively 


Selected Problems and Lessons: 

• External corrosion, internal wear, and corrosion fretting of 
RSB and BF actuators. 

Perform periodic external and internal 
maintenance inspections on all critical hardware 
for corrosion damage. 

Establish a fleet leader inspection plan to assess 
the condition of the hardware over time and 
uncover any unexpected issues. 

Apply corrosion-preventive primer (e.g., Super 
Koropon) to all external hardware surfaces. 



ACTUATOR ACTUATOR FCU ACTUATOR ACTUATOR 



Rudder/Speedbrake and Body Flap actuation systems 



Fretting wear 


Unit#1 outboard planetary. S/N D7C021 


Fretting fatigue 


Multiple bands 
imply multiple 
contact positions 
with low amplitude 
dither motions 


RSB actuator planetary gear corrosion damage. 
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Mechanical Flight Controls (MFC) Lessons 


Problem Description 


Lesson 


RSB PDU gearbox discrepancies: gear 
scuffing, dark/contaminated/low oil level. 


RSB and BF actuators internal grease 
degradation (Braycote 601) 



GEARBOX 





Perform periodic inspections on isolated fluid cavities that 
normally are not checked/sampled and are not continually 
filtered, e.g., gearboxes. 

Utilize grease with extreme pressure (EP) additives 
necessary for efficient boundary lubrication, better out- 
gassing qualities, more stable base oil, corrosion-inhibitors 
and improved properties against physical separation and 
chemical breakdown. 


Assure correct alignment of interfacing seals and close- 
tolerance components to prevent leakage, e.g., transfer tubes. 
Build special alignment tooling if necessary. 

Conduct post-assembly leakage tests, particularly of internal 
components with no available insight post- assembly. 
Conduct periodic inspection of critical hardware cavities 
such as gearboxes. 


Body Flap PDU transfer tube 
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Main Propulsion System (MPS) Lessons 


MPS Subsystem Overview: 

• Consists of the ET, Orbiter MPS, and Space Shuttle Main Engines 
(SSMEs). 

• Extensive GSE also exists to service, monitor, and maintain these 
elements. 

• Tasked with storage, conditioning, distribution, and combustion of 
cryogenic LH2 and L02 propellants to provide first and second stage 
thrust for achieving orbital velocity. 


Selected Problems and Lessons: 



High Point Bleed QD 

LH2/He Prepress Port 
Helium Fill Port 

Fill & Drain QD 
Fuel Bleed 
Prestart 

Conditioning System 
Helium Tanks 


High-Press LOX Turbo-Pump 
Low-Press LOX Turbo Pump 


LOX Umbilical Assembly 


Surge Tanks 


Engine Purge QD 


LH2 Feed Line 
Disconnect 
LH2 Umbilical Assembly 


Low-Press Fuel 
Turbo Pump 


LOX/Hellum 
Pre-press QD 


• LH2 feedline flowliner cracks. MPS Hardware in the Orbiter Aft Compartment 

Fully characterize the environment, especially across element 
interfaces where relatively little may be known. 

Unintended consequences of fluid/acoustic environment, 
untested during certification. 

Avoid use of stamped flowliner slots, favoring machining if 
required at all, especially where debris liberation is hazardous. 

Proved success of “inspect and fly” methodology. 

• Flow control valve poppet break. 

Even structure with very high natural frequencies can be 
excited with acoustic inputs. 

Inspection methods need validation by complementary NDE, 
especially when cracks are very tight due to material properties 

and/or driving environment. STS - 126 broke " rcv poppe ' 
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Main Propulsion System (MPS) Lessons 


Problem Description 


Lesson 


L02 feedline BSTRA ball cracks. 







OV103 Cracked BSTRA Ball 


Position indicator mounting location. 






Engine cutoff system investigation. 





COPV limited life due to stress 
rupture. 



Problems with hardware certified by 
similarity to other hardware in a 
“daisy chain”. 






Casting process can cause voids/inclusions in the parent material, leading 
to thermally-induced crack formation during subsequent steps in the 
manufacturing process or while in service. The HIP manufacturing 
technique can reduce or eliminate voids in the cast material as it cools. 

An inadequate acceptance testing and inspection process failed to detect 
the small cracks. This screening process should be made as robust as 
possible. 

Valve position indicators should be installed downstream of the propellant 
valve element rather than upstream at the actuator or in the drive train. 

Use staged pressurization to reduce peak temperature seen during helium 
loading operations, greatly increasing life. 

Important to re-evaluate the original design assumptions with flight unit 
performance test data as it becomes available. 


The approach of going after “low hanging fruit” does not always save 
money or time. 

The most likely, most dangerous cause should be investigated first. 
Common cause failures are very real and can seriously degrade system 
reliability. 

Avoid certifying hardware by similarity to other hardware which was 
itself certified by similarity unless a very good basis of testing is known. 
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Main Propulsion System (MPS) Lessons 


Problem Description 

Lesson 

Oxygen material compatibility. 

Use of Inconel and Monel was determined to be preferable to stainless 

M % 

steel based on the reduced propensity to ignite upon particle impact. 
Testing should always be performed in a representative G02 environment 
to ensure that design sensitivities are identified early. 

Careful consideration on use aluminum or aluminum alloys for piece- 

Oxygen 

parts that experience relative motion in a G02 or L02 system. 

Gaseous helium check valve 
poppet jamming . 

Careful attention should be paid to the Length-to-Diameter (L/D) ratio, 
radial clearances, and material selection when designing a sliding 
interface between valve piece-parts. 

Perform representative testing, including high flow demand. 

Sharp edges and comers on 
component piece-parts. 

Use generous radii at edges along sliding surfaces, at neck-downs, and on 
internal comer to prevent binding/jamming and reduce stress 
concentrations. 

Pre-valve main seal cracking due to 
slamming. 

Design control preferred over operational control. 

Potential for inadequate initial 
certification and of program 
requirements “creep”. 

Periodically perform a review of environments for impact on hardware 
certification. 

Use field and flight failure records as a guide. 

Designed-in features or analytical 
margin are frequently not certified 
by test. 

Perform burst testing, verification of redundancy, or operation with off- 
nominal inputs as required to verify design. 
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OMS and RCS Lessons 


OMS/RCS Subsystem Overview: 

• Pressure fed hypergolic propulsion system using monomethylhydrazine 
(MMH) as the fuel and nitrogen tetroxide (NTO or N204) as the oxidizer. 

• OMS provides orbit altitude adjustments for the space shuttle vehicle 
during the orbit phase, orbit insertion after ascent phase (OMS-2 bum), 
and the deorbit retrograde bum for the entry phase of flight. 

• RCS is located in the two OMS pods and the forward module of the 
vehicle. 

^Provides the shuttle with orbit attitude control using multiple 
thmsters and entry control until aero surfaces can provide sufficient 
control authority for the shuttle. 

Selected Problems and Lessons: 

• PRCS Pilot Operated Valve fuel valve seal extmsion. 

A properly maintained hypergolic system can mitigate or nearly 
eliminate nitrate related ground and in-flight failures. 

Need integrated hardware qualification testing with propellant, 
moisture and time exposure. 

Implement preventative maintenance schedule to minimize in- 
flight failures by flushing all thmster valves periodically. 
Implement thmster chamber moisture control during ground 
operations. 

Reduce low-level oxidizer leakage at the poppet/seat interface. 
Improve the corrosion resistance of materials at the poppet/seat 
interface. 

Minimize contact area between mating parts/surfaces to decrease 
potential nitrate-related stiction. 


Primary 
Thruster (14) 



Components 


Forward RCS Module and OMS/RCS pods 



Iron nitrate on RCS 
POV pilot seat assembly 
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OMS and RCS Lessons 


Problem Description 

Lesson 

PRCS Pilot Operated Valve fuel valve 
seal extrusion. 

& Thmster chamber moisture control hardware needs to provide sufficient 
ventilation of oxidizer vapor through use of a purge system. 

Sometimes configuration changes, e.g., Universal Throat Plug Assembly 
(UTPA), can bring unintended consequences. 

Thmster valve leak rate tolerances should be small. 

Use GN2 flow rate test to screen for failure. Consider developing current 
trace as part of screening method. 

Significant benefit of establishing a dedicated team (i.e., Tiger team) to find 
root cause. 

PRCS combustion instability. 

STS-68 POV Extruded PTFE (Teflon) 

Although generally stable, instabilities were induced when helium bubbles 
injected into the fuel stream (propellant saturated with ullage). 

Add stability screening test to thmster hot- fire ATP. 

Add wire wrap as protection for thmster instability. 



FORP and Pc sense tube bum through 
resulted in near catastrophic failure on 
STS-51. 

Improve Pc tube design, e.g., avoid “S” shape or increase tube thickness. 
Develop Pc tube flushing schedule to remove FORP. 

Do not fire a suspect thmster during flight unless conditions are well 
understood and covered by qualification testing. Rather, inhibit and 
reprioritize thmster unless needed as a last resort. 

Avoid very short firings; smaller impulse bit equates to less efficient 
combustion and more FORP formation. 
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OMS and RCS Lessons 


Problem Description 

Lesson 

Vernier thruster reboost results in 
valve overtemperature. 

Restrict vernier usage to specific reboost duty cycles. Develop go-no-go 
duty cycle criteria for long duration reboost requirements. 

Primary RCS thruster L1L injector 
failure in ground test. 

Avoid the unique set of ground test conditions that can lead to intra- 
manifold explosions such as: leaking thruster valves, leak duration, 
horizontal configuration, ambient pressure and thermal conditions. 

Thrust chamber chips (R-5 12 
discilicide coating mechanically 
induced spalling). 

Perform periodic inspections and track growth of known chips. 

Injector film cooling provides longer primary thruster chamber life. 
Characterize chips and develop acceptance rationale such that thruster 
nozzles do not need to be R&R for every chip. Columbium nozzles with 
disilicide coating is more robust than initially understood. 

OMS quantity gauging fuel probes 
problems. 

fuel 

tp 

Have altemate/redundant quantity gauging methods, e.g., ground loading 
gauging, in-flight engine firing time integration, PVT calculation. 
Manufacturer’s expertise was with capacitor gauging sensors and not 
with complex electronic systems (totalizer). Recognize manufacturer’s 
expertise and limitations. Need right expertise for right technology. 

More thorough acceptance test to screen out deficiencies. 

Ground operation issues: 

High ground processing timeline, 
cost, hazards (hot R&R, fire), 
supportability and obsolescence. 

Closely track removal rates, repair times, spares counts, limited life 
hardware, and status of repair agencies. 

Implement Integrated vehicle Health Monitoring system (IVHM). 
Relocate system components to the pod outer mold line and/or add quick 
component access doors to avoid the need to remove the whole pod for 
troubleshooting. 

■0 Few area heaters are far easier for ground checkout. 
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Fuel Cell and PRSD Lessons 


Fuel Cell/PRSD Subsystem Overview: 

• 3 fuel cell powerplants on each Orbiter capable of providing 2- 
10 kW nominal power (16 kW max) with voltage regulation 
between 27.5 - 32.5 VDC. 

• Low temperature alkaline fuel cell (AFC) power plant consists 
of a 96-cell stack and an accessory section. 

• Reactants (oxygen and hydrogen) are stored in four or five 
Power Reactant Storage Distribution (PRSD) tanks, also 
located in the mid body of the Orbiter. 


Selected Problems and Lessons: 

• STS-2 FC1 shutdown following H2 pump/water separator 
failure and FC flooding. 

3£ Bus-tying the suspicious fuel cell with a healthy fuel 
cell is the method for monitoring performance of the 
suspect fuel cell when one fuel cell picks up the load 
from another. 

■£7 For vital systems, redundancy in monitoring the 
system’s health is as important as it is in the system 
itself. 



Orbiter Alkaline Fuel Cell Powerplant 



Hydntfin pump 
liparato/ 

Ha fmiin 


H?{**t*r 
vapor From 
csll stack 


& 




Fuel Cell Hydrogen Pump Separator; 
aspirator tubes highlighted in red 


21 



Fuel Cell and PRSD Lessons 


Problem Description 

Lesson 

STS-4 condenser exit temperature 
downward shift. 

Validate all instrumentation at maximum loads by running the fuel cells at 
high power for approximately 30min shortly after fuel cell activation. 

STS-43 post-landing: fuel cells left 
on bus without water removal 
capabilities. 

Operation timelines and procedures must be developed, executed and 
verified to avoid potential permanent damages to systems. 

Cells can provide power even when Fuel Cell is off. 

Fuel Cell shutdown/restart in mid- 
flight. 

Redundant power systems need to be designed and proven in a space 
environment, i.e., shutdown/restart capability. 

STS-83 FC shutdown due to 
perceived crossover. 

When vital instrumentation has a history of being erratic, a redundant 
measurement (such as FCMS data) must be available to rule out failures. 
When failure can neither be proved, nor disproved in-flight, it is better to 
perform sating procedures and eliminate the risk of an unplanned failure. 

Port plugging (inability to clear inerts 
due to eel) corrosion buildup]. 

All materials within the system need to be indentified for potential particles 
release and must include Failure Mode and Effects Analysis. 

Carbonate formation inside 02/H2 inlet/outlet ports can arise from 
carbonation of fiberglass/epoxy FC frame. 

Importance of maintaining system cleanliness. 

Port plugging can be discovered during N2 diagnostic testing. 

Can still fly with some port plugging so long as performance is monitored 
and purges are performed when necessary. 

STS-l 15 loss of phase in coolant 
pump and H2 pump. 

Three-phase electrical motors need to be evaluated and tested for being 
able to operate in two-phase. 

Additional purges were necessary to maintain 2-phase operation. 


Fuel Cell and PRSD Lessons 


Problem Description 

Lesson 

Water relief line freeze possibility. 

Operating environments need to be identified, implemented in design, 
performed and evaluated. 

When new mission configurations arise (e.g., docking to a station), new 
problems may also arise which might require re-evaluation of risks. 

H2 pump current sensor sensitivity. 

Unusual sensor activity can indicate a trend toward failure of the H2 
pump motor, and thus a subsequent loss of the fuel cell. 

When using an instrumentation measurement as a failure indication, 
make sure that all causes for output variations are understood and 
procedures are in place for filtering out actual failure indications. 

< 

( 

STS- 127 long sustaining heater 
:ycle. 


Effects of thermal environments must be included in design and 
demonstrated by testing. 

Due to lower power requirements on SSPTS flights, the fuel cells 
sometimes require heating to return to operating temperature. 

When the fuel cells are performing on the low end of their certification 
(2kW), sustaining heater cycles become a necessary component for 
operation: if they fail ON, the fuel cell needs to be safed immediately if 
the coolant system malfunctions in any way; if they fail OFF, the fuel 
cell load needs to be increased to maintain temperature, resulting in a 
cryogenic margin impact. 

Multiple 02/H2 flowmeter erratic 
readings and failures (shifts in 
calibration, zero output, full-scale 
output, constant output). 

Flow path must be external to the electronic & instrumentation housing. 
Sometimes, the cost and time required to redesign a troublesome 
component outweighs the benefit. 

So long as a flowmeter still accurately indicates a purge, it will also 
accurately indicate a significant leak. 


EPD&C, Wiring and Insulation Lessons 


Subsystem Overview: 

• Electrical Power Distribution and Control (EPD&C) 
subsystem consists of a 3 -bus system that distributes electrical 
power (24V to 32V) from the fuels cells to the forward, mid, 
and aft sections of the Orbiter. 

• EPD&C system also includes three 120V, 400Hz, 3- 
phase, alternating current (AC) power busses, generated 
by sets of 750volt-ampere (VA) inverters from the main 
busses. 

• Electrical Wiring and Insulation (EW&I) is made up of all 
electrical wiring and connectors external to subsystem boxes. 

• Approximately 600 harnesses, 7,000 connectors, 
120,000 wire segments and 228 miles of wire. 



Orbiter electrical system 


Selected Problems and Lessons: 

• STS-93 wire short description: 

• Short circuit in the AC system during ascent caused 
the loss of 2 main engine controllers. 

• Post flight investigation found the cause to be a 
damaged wire with an exposed conductor that 
contacted a nearby burred screw head. 

• As a result of the inspections, many other incidents 
were found of damaged wires and improperly 
protected wire harness. 



Burred screw 


STS-93 wire short 
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EPDC, Wiring and Insulation Lessons 


Selected Problems and Lessons: (Continued) 

• STS-93 wire short (Lessons): 

Ground maintenance and vehicle modifications can 
cause damage like burrs on screw heads that lead to 
later wire damage. 

Closeout inspections did not focus on inspecting the 
wiring. 

Use auxiliary lighting and mirrors to inspect all 
accessible wiring. 

Formalized and more clearly defined wire inspection 
criteria and processes. Perform periodic wiring 
inspections; particularly inspect for previously 
undetected wiring discrepancies. 

Any primary wire insulation damage is 
unacceptable regardless of whether conductor is 
exposed or not. 

Separate critical redundant wire runs and/or install 
barrier materials to protect against the effects of arc 
tracking. 

& Widespread use of chafe protection greatly reduced 
wire damage. 

Enhanced workforce awareness and training is 
paramount in ensuring the health of electrical wiring. 



Orbiter wiring 



Example of arc tracking 
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EPDC Wiring and Insulation Lessons 


Problem Description 

Lesson 

Vehicle wire damage due to 
ground maintenance activity 
(mechanically induced damage). 

Important to perform specific detailed closeout inspection after work on cables or 
connectors. 

Wiring is easily damaged by nearby work activity. 

Training of personnel is critical both to prevent damage and to detect damage. 
Wire chafe protection is important. 

STS-4 Heater fuse failure. 

Repetitive thermal cycles on a fuse can cause it to mechanically open 
Space and usage environments must be considered for fuse de-rating. 

Circuit breaker problems. 

Cycling circuit breakers recovers function. 

Debris is created from internal wear on circuit breakers. 
Avoid use of CBs as power switches and redundant controls. 

Personnel switch bumps. 

Redundancy of controls and inhibits important for critical functions. 
Utilize switch guards. 

Lightning monitoring system. 

Low frequency bus voltage monitoring systems are inadequate for detecting 


lightning transients on vehicle power busses. 

In order to detect lightning transients, monitoring system frequency response 
should be higher than lowest resonant frequency of vehicle power bus. 

Low level signals easily couple onto bus from distant strikes, several miles away. 
-$■ Use pulse energy (vs. voltage magnitude) to assess damage (Wunsch-Bell). 
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Pyrotechnics Lessons 


Subsystem Overview: 

• Explosive devices that contain energetic mixtures of fuel and oxidizer 
compounds that when activated by electrical or mechanical initiation 
converts chemical energy to mechanical energy. 

•50 Government Furnished Equipment (GFE) pyrotechnics devices have 
been certified for human spaceflight by JSC 

• These include the NASA Standard Initiator (NSI), NASA 
Standard Detonator (NSD), several types of power pressure 
cartridges, mild detonating fuse and detonating cords, shear bolts, 
explosive bolts and frangible nuts, guillotines, reefing line cutters 
and other devices. 

• About 140 NSIs flown on each Shuttle flight. 

• Over 13,750 NSI fired during flight on the shuttle vehicles 
without a failure, with a reliability of 0.9999 at a confidence level 
of 95%. 

Selected Problems and Lessons: 

• STS-8 ejected Nose Landing Gear Extension Thruster. 

Verify that hardware design and qualification testing account for 
all potential flight conditions. 

Never assume contained air pressure is available as an assist to 
pyrotechnic performance. 


NLG extension thruster and likely trajectory of 
piston and wheel on STS-8 landing 
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Pyrotechnics Lessons 


Problem Description 


Lesson 


3!4 inch frangible nut skew firing. 


Propellant powder deterioration. 


Inconel alloy 718 became stronger 
and harder to break. 


Pyro-valve poppet failure. 





Be sensitive to the firing skew present in all redundant pyrotechnic 
systems. 

Changes in the manufacturing processes had occurred that were not 
evident during the commercial certification of the material. 

Be aware of raw material processing “refinements” over time which 
occur to satisfy other primary material users. Need for control of material 
grain sizes (e.g., roll milling). 

Need to specify both minimum and maximum material properties, 
particularly in applications that are sensitive like separation systems. 

Debris in bore and/or manufacturing dimensional variations can cause 
pyrovalve to not properly seal after firing/actuation. 

The NSI is an initiator and should not be used as a pressure cartridge, 
i.e., use NSI plus booster pressure cartridge. 


Pyro valve (post-fire) showing result of 
failure. 
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Summary/Conclusions 


A selected few “key” lessons in Orbiter Power and Propulsion subsystems were presented. 

• Subsystems included: APU, Hydraulics, WSB, Mechanical Flight Controls, MPS, OMS/RCS, Fuel 
Cells/PRSC, EPD&C, Electrical Wiring and Insulation and Pyrotechnics. 

• These lessons cover the nearly 40 year history of the Space Shuttle Program. 

There are several common lessons across subsystems. These include: 

ft- Design solutions are preferred over continuous maintenance. However, verification of the validity 
of operational limitations is preferred over re-design, if proven acceptable, 
ft Importance of a full understanding of the operational environment. 

ft Importance of a thorough and integrated mission-representative qualification test program, 
ft System improvements, corrective actions for other problems, process changes, and/or mission 
requirement changes can bring unintended consequences or unexpected sensitivities, 
ft Implement a robust program-wide contamination control plan. 

ft Importance of a robust and thorough Corrective Action process that avoids extensive latitude for 
accepting repeated failures without preventive corrective actions, 
ft Important to design critical systems to be fail-safe and that redundant elements are independent, 
ft Establish periodic and preventative maintenance, 
ft Establishing a fleet leader test program. 

ft Trending of hardware operational flight and ground test data to uncover any out-of-family or 
degraded performance, well before they become critical, 
ft Always verify the system integrity after any intrusive work is performed on the vehicle, 
ft Design systems for ease of leak checking and operational verification. 


It is the hope of the authors that these lessons are useful to future space programs. 
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